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ABSTRACT 

Designed to help trainers of military maintenance 
technicians evaluate the cost-effectiveness of using maintenance 
simulators as opposed to using actual equipment trainers, this report 
documents the value of using the synthetic training devices to 
duplicate the performance characteristics of operational equipment 
under both normal and many malfunction conditions. The report also 
outlines an evaluation of 12 studies conducted since 1967 which show 
that student achievement in courses that used maintenance simulators 
was either the same or better than those in courses using actual 
equipment trainers. While it is noted that the acquisition cost of 
maintenance simulators was 'typically less than that of actual * 
equipment trainers, the report cautions that this finding must be 
qualified by the fact that effectiveness is based primarily on school 
achievement rather than on-the-job performance, and cost figures are 
based primarily on acquisition rather than on life cycle costs. 
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ABSTRACT 

The cost-effectiveness of maintenance simulators, compared 
to actual equipment trainers, is evaluated for training military* 
maintenance technicians. Maintenance simulators are as effective, 
as actual equipment trainers when measured by student achievement 
at school; there is. no difference in the job performance of stu- 
dents trained either way, according to supervisors' ratings, in 
only one study. The acquisition cost of maintenance simulators 
is typically less than that of actual equipment trainers. The 
cost to develop and fabricate one unit of a simulator was less, 
than 60 percent of the cost of its counterpart actual equipment 
trainer in 7 of 11 cases investigated. The cost of fabricating 
an additional unit of the simulator was less than 20 percent of 
the cost of its counterpart actual equipment trainer in 9 of 
these 11 cases. Acquisition and use of a maintenance simulator 
over a 15-year period would cost 38 percent as much as an actual 
equipment trainer, according tp the only life-cycle cost compari- 
son that has been reported, since maintenance simulators and 
actual equipment trainers are equally effective and since main- 
tenance simulators cost less, it is concluded that maintenance 
simulators are cost-effective compared to actual equipment 
trainers. This finding is qualified because it is based on a 
limited number of comparisons^ because effectiveness is based 
primarily on school achievement rather than on the job perform- 
ance, and because it is ba^ed primarily on acquisition rather 
than on life-cycle costs. 
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SUMMARY 



A. PURPOSE 

This paper evaluates the 0 cost-ef fectiveness of maintenance' 
simulators, compared to actual ifjuipmen-t trainers, for traininq 
military personnel to maintain military equipment.* 

B. BACKGROUND ■ „ 

Maintenance simulators are synthetic training devices that 
appear to duplicate the per5ormance characteristics of opera- 
tional equipment under normal and many malfunction conditions. 
"Actual equipment trainers are operational equipments that are 
provided with powe^r, inputs,' and controls needed to make them 
operate in a classroom. Maintenance simulators incorporate some 
type of computer support to provide a large variety of malfunc- 
tions for instructional purposes, are designed to withstand' 
abuse .in a classroSm., 'do not "expose students to dangerous condi- 
tions, and can measure student performance for the information 
of both students and instructors. They are generally less 
expensive to procure than actual equipment trainers. Actual 
equipment trainers provide students an opportunity to train^on 
the actual equipment they will be expected to maintain aft^r 



*In 1976, the Defense Science Board recommended 'cost-effective- 
ness evaluations of military training. This study is one of 

„ several undertaken in response to that recommendation, the 
study was performed for the Office of the Deputy Under Secre- 
tary of Defense for Research and Engineering (Research and - 
Advanced Technology), under the -technical cognizance of the 
Military 'Assistant for Training and Personnel Technology. 



they leave school. Limitations of such trainers are that, 
being designed for operational rather than instructional 
purposes, they may break down and be difficult to maintain in 
a classroom setting. They provide only limited opportunities 
for demonstrating malfunctions because instructors must install 
"faulty" components, which always takes some time and may be 
. inconvenient. Actual equipment trainers do not include facili- 
ties for measuring student performance; this would require a 
complex and costly process of redesign; 

C. EFFECTIVENESS 

The effectiveness of maintenance simulators has been evalu- 
ated in 12 studies conducted since 1967* These . evolved mainte- 
nance training for equipment used in sonar avionics, radar, 
propellers, flight control, navigation, aircraft power plant, 
communication, and ship automatic boiler control systems. Stu- 
dent achievement in 12 courses that use'j maintenance simulators 
was the same as or better than that in comparable courses that 
used actual equipment trainers; in one case, student achievement 
with a maintenance simulator was less. In one case where on-the- 
job performance, was evaluated, supervisors* ratings showed that 
there was no difference between students trained with a simulator 
or an actual equipment trainer-. Students trained with mainte- 
nance simulators completed their courses in less time than did 
those who used actual equipment trainers. In three cases where 
such data were collected, time savings were 22, 50, and 50 per- 
cent, respectively. Most students who use maintenance simulators 
have favorable attitudes toward their use; instructors are split 
about equally in having favorable, neutral, or negative attitudes 
toward the use of these simulators. • 
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D, COST' 

Maintenance simulators appear to cost less to acquire than 
do actual equipment trainers* The cost to design, develop, and 
fabricate one unit of a simulator is less than 60 percent of 
the unit cost of its counterpart actual equipment trainer in 7 
cases out of a sample of 11; in the remaining four cases the 
simulators cost more than the actual equipment trainers* Once 
developed, the cost of fabricating an additional unit of a 
simulator is less than 20^ercent of the unit cost of its 
counterpart actual equipment trainer in 9 of those 11 cases; 
in only one case did the simulator cost more to fabricate than 
the actual equipment trainer* 

In the one available case of a life-cycle cost-effective- 
ness evaluation, the Air Force 6883 Test Stand Three-dimensional 
Simulator was as effective as the actual equipment trainer, both 
at school and on the job. The total costs for the $ame student 
load over a 15-year period were estimated to be $1.5 million 
for the simulator and $3.9 million for the actual equipment 
trainer; that is, the simulator would cost 38 percent as much 
to buy and use as would the actual equipment trainer. 



CONCLUSIONS 



Conclusion 1 . Maintenance simulators are as effective as 
actual equipment trainers for trainings mi litay personnel, as 
measured by students' achievement at school and, in one case, 
on the job. The use of maintenance simulators saves some of 
the time needed by students to complete courses, but data on 
this point is limited. Students favor the use of maintenance 
simulators; instructors are favorable, neutral, or negative 
toward the use of simulators in about equal numbers. 

Conclusion 2 . The acquisition cost of maintenance simula- 
tors are J:ypically less than that of actual equipment trainers. 
The cost to develop and fabricate ore unit of a simulator was 
less than 1 60 percent of the cost of actual equipment trainers 
in 7 of 11 cases examined; the cost to fabricate an additional 
unit of a simulator was less *-han the 20 percent of cost of 
actual equipment trainers in 9 of the 11 cases. The one avail- 
able life-cycle cost estimate shows that purchase and use of a 
simulator would cost 38 percent as much over a 15-year period 
as it would to buy and use an actual ( ^uipment trainer. 

Conclusion 3 . Maintenance simulators are as effective as 
actual equipment trainers for training maintenance personnel. 
In addition, they cost less to acquire. Therefore, maintenance 
simulators arc cost-effective when compared with actual equip- 
ment trainers. 

Conclusion 4 . In general, the data on the cost and effec- 
tiveness of maintenance simulators have not been collected sys- 
tematically. Therefore, there is no basis at present for mak- 
ing trade-offs between the effectiveness and cost of different 
types of maintenance simulators on such issues as two-dimensional 



versus three-dimensional design, the complexity of maintenance 
simulators (in such terms as number of malfunctions and instruc- 
tional procedures), the extent to which simulators should pro- 
vide a mixture of training in general maintenance procedures 
and/or for maintaining specific equipments, and the optimum 
combination of maintenance simulators and actual equipment 
trainers for training technicians at school. 

There have been insufficient studies on the amount of stu- 
dent time saved with the use of maintenance simulators. There 
have been no studies on whether the use of maintenance simula- 
tors influences the amount of student attrition at school. 
There have been no studies to collect objective measures of 
performance of maintenance technicians on the job after train- 
ing, either with simulators or actual equipment trainers. 

Conclusion 5 . Maintenance simulators now under development 
have not yet taken advantage of recent technological advances 
such as videodiscs, automated voice input and output, and minia- 
turization sufficient to make them readily portable. Reductions 
in size would make it possible, as well as more convenient, to 
use maintenance simulators for refresher training near job sites 
and for performance evaluation and/or certification of mainte- 
nance personnel on an objective basis in operational environ- 
ments. Extreme reductions in size would make it possible to 
use maintenance simulators as job aids in performing maintenance 
on operational equipment, thus assuring a close link, not yet 
available, between facilities used for training at school and 
for performance on the job. 
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RECOMMENDATIONS 



Recommendation 1 . Collect data to enlarge what is now 
known about the effectiveness l maintenance simulators and 
actual equipment trainers at school* Data are needed on speci- 
fic knowledge and skills acquired by students at school, the 
time needed to complete courses, attrition rates, and instruc- 
tor attitudes toward the use of simulators and actual equipment 
trainers . 

Recommendation 2 . Collect objective data on the performance 
of technicians on the job after training with simulators or 
actual equipment trainers. Determine the transfer of training 
of maintenance skills from school to the job, when either 
maintenance simulators or actual equipment trainers are used 
in training courses . Such data should be collected in a way 
that will permit a. deter mina ti on o f the rel at ive effectiveness 
of maintenance simulators with varying characteristics such as, 
types of design, degrees of complexity, physical appearance, and 
in generic and specific maintenance training courses. 

Recommendation 3 . Collect cost data in sufficient detail 
to permit the development of cost-estimating relationships for 
maintenance simulators. The Cost elements should account for 
all portions of the total costs incurred .to procure and use 
maintenance simulators and actual equipment. A suggested struc- 
ture for the collection of procurement cost data is contained in 
this paper. 

Recommendation 4 . Design and conduct studies of training 
wji t h __mai n t e n a nee s j. m u la tors and actual .equipment ^tjrainers, 
that will yield trade-offs between the level of effectiveness 
and total cost as functions of the characteristics of training 



equipment, the ways it is ur.ed, and the types of training 
involved. 

Recommendation 5 * Develop a procedure to categorize the 
functional characteristics of maintenance simulators and actual 
equipment trainers in ways that will relate to their effective- 
ness for training. 

». Recommendation 6 . Develop objective measures of the job 
performance of maintenance personnel in operational settings to 
provide valid measures with which to evaluate the effectiveness 
of s\nnlators and actual equipment trainers. 
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I ♦ INTRODUCTION" 

A ♦ PURPOSE 

The purpose of this paper is to evaluate the cost-effec- 
tiveness of maintenance training simulators, compared to actual 
equipment trainers, for use in training military personnel to 
maintain operational equipmenti Both types of equipment have 
been used at technical training schools to train personnel to 
perform* corrective and preventive maintenance at organizational 
and intermediate levels. 

r 

Actual equipment trainers have long been used in technical 
training schools for two significant reasons: (1) they can be 
acquired simply by ordering additional unitr of operational 
equipment already being procured for use in weapon and support 
systems; and (2)^ they provide realistic training on the equip- 
ment to be maintained after the student leaves school. Opera- 
tional equipment is often modified for training purposes by, 
for example, placing it on a stand and adding power supplies, 
input signals, and controls needed to make it operate in a 
classroom. There has been a trend, in recent years, to use 
maintenance training simulators rather than actual equipment 
for training purposes. Maintenance simulators are said to have 
advantages for use in training such as lower cost, ability to 
demonstrate a wider variety of malfunctions, and more freedom 
from breakdown in the classroom. These advantages will be 
considered later in this paper. 

The purpose of maintenance training is, obviously, to 
train personnel to maintain complex equipment; this requires 
both technical knowledge and manual job skills. Maintenance 



training familiarizes the student with the layout of the equip- 
ment, sources of power, usfe of tools and test equipment, Safety 
requirements, control settings, instrument readings, operating 
procedures, and the like. Maintenance personnel must be able 
to diagnose malfunctions; identify, replace, or repair faulty 
components; verify that all components perform within prescribed 
tolerances; and perform tests to insure that the entire equip- 
ment has been returned to working order. This type of training 
can be provided by a variety of means such as conventional class- 
room instruction, studying technical manuals, learning fault- 
finding procedures by self-study, computer-ass iated or computer-^ 
managed instruction and, of course, the use of various types of 
training devices. The issue addressed in this paper is whether 
maintenance simulators are more cos'tef f ective than actual equip- 
ment trainers for training military maintenance personnel. 

Even if maintenance simulators are more cost-ef fecitve at 
school for training personnel, it is obvious that traininq is 
supposed to prepare technicians to maintain operational equip- 
ment in the field and not just to perform well at school. Thus, 
the major substantive issue is to compare how personnel trained 
with maintenance simulators or actual equipment trainers actually 
maintain operational equipment in the field. Whether this ques- 
tion can be answered on the basis o[ currently available infor- 
mation is considered later. 

» 

B. TYPKS AND LOCATIONS OF MAINTENANCE ACTIVITIES 

There are two main types of maintenance activities: 
1. Corrective ma i ntenance applies to equipment that has 
failed or is known to be operating improperly. In the 
typical case, a malfunction is noted and reported by 
operational personnel who use the equipment and re- 
paired by the maintenance personnel. Corrective 
maintenance involves troubleshooting, diagnosing the 
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reason for a malfunction, identifying the component 
(or components) that has failed, repairing\and/or 
replacing the faulty component (or larqer module of 
which it may be a part) and, fifally, testing\md 
calibrating to assure that the malfunction no lortger 
exists • 

2. Preventive maintenance applies to equipment that has. 
not failed and appears to be operating properly. it 
involves periodic inspection, cleaning, testing, and 
calibrating of equipment; this may include the replace- 
ment of functioning parts in^ accord with schedules 
established to reduce the possibility of future break- 
downs. 

To be effective, both types of maintenance require not 
only proper training but also proper tools, test equipment, 
relevant and up-to^cfate technical documentation, and efficient 
diagnostic procedure?; the equipment itself must be designed 
to permit convenient access, test, repair, and replacement of 
parts; and there must also be a proper supply of spare parts 
and an adequate number of maintenance personnel, including 
supervisors, to handle the workload. 

Maintenance activities are also associated with the places 
where they occur. There are three types, as described below: 
1. Organizational maintenance is performed on equipment 
on the flight line or in the field by maintenance 
personnel assigned to the unit that operates the equip- 
ment. It consists generally of inspecting, servicing, 
lubricating, adjusting, and replacinq faulty assemblies 
and subassemblies (line-replaceable units or LRUs)\ 
2 * Intermediate maintenance is performed in maintenance - 
'Shops by personnel assigned to a base or support drgani- 
zation. It generally consists of calibration, repair 
or replacement of damaged or unserviceable parts, bhe - 
emerqency manufacture of nonavailable parts, and pro- 
viding technical assistance to the using organization . 
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3. D epot maintenance is performed at a central, industrial- 
, type facility and consists of large-scale repair, mod i- 
ficjation, and refurbishment* 

C. MAGNITUDE OF THE PROBLEM 

The Department of Defense, as of 1976, possessed weapon > 
systems and equipment which cost about $125 billion to acquire 
(see Table 1). About $49 billion was requested for procurement 
in FY 1982 (Brown ^1981, p. 212). The purpose of maintenance is 
to keep these weapons and their support equipment in a state 
of operational readiness to meet mission requirements and to do 
this in a timely and economic manner. Maintenance is a criti- 
cal aspect of defense planning and operations and coses $18-20 
billion each year, including the costs of spare parts, supplies, 
and modifications (Turke 1977; p. 5). 

TABLE 1. ACQUISITION COST OF WEAPON SYSTEMS ANO EQUIPMENT IN 
USE OR ASSIGNED, JULY 1, 1976 (TURKE, 1977) 





Acquisition Cost 


Billions of 
Dollars 


Percent 


Mil itary Department 








Army 


19. 


2 


15 


Navy 


61. 


7 


49 


Air Force 


45. 


3 


36 




126. 


2 


100 


Weapons Group 








Aircraft 


54 


1 


43 


Ships 


38 


8 


31 


Missiles 


8 


7 


7 


Vehicles 


9 


0 


. 7 


Other 


15 


6 


12 




126 


2 


100 
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Accordina to the General Accounting Office, the Army spends 
25 percent ($7.0 billion in FY 1978) of its annual budget on 
maintenance; over 200,000 mechanics and equipment operators in 
the Army have specific unit-level maintenance responsibilities 
(GAO 19 78, p. 1). m the Air Force, maintenance requires about 
28 percent of the work force (military arid civilian) and costs 
betweep $5 and $7 bi 1 1 ion annual'ly" (Townsend 1980). Labor for 
repairs is estimated to account for 39 percent of the cost of 
recurring logistical support (Fiorello 1975). Training is only 
one of many "factors that influence effective maintenance, e.g., 
design of equipment to assure high, inherent reliability; design 
of equipment to permit unambiguous identification of failed 
components; easy access for test and replacement of components; 
the availability of spare parts and test equipment; up-to-date 
technical documentation, tools, job aids, and the like.* 
Specialized skill training at military schools costs about 
$2.9 billion or 33 percent of the cost of individual training 
each year (Department of Defense, Military Manpower Training 
Report (MMTR) for FY 1981, p. 6]; the portion attributed solely 
to maintenance training is not known. 

High turnover among enlisted personnel increases the diffi- 
culty of maintaining military equipment. According to planning 
estimates for FY 1981, about 337,000 personnel were to be re- 
cruited; 31 3,000 (93 percent) of these were expected to completj? 
recruit training; only 64,000 (37 percent) would reenlist for 
a second term [MMTR FY 1981, p. m-3; estimate on reenlistment 
from all volunteer force data base, asd (MRAsl), 20 Mar 1980]. 



See Integrated Technical Documentation and Training (I TOT) 
(1978) and Navy Technical Information Presentation Program 
(NTlPP) (1977) for a review of current efforts to improve- 
technical documentation required for maintenance; see Rowan 
(1973) and Post and Price (1973) for recent reviews of studies 
which compare performance of maintenance technicians using 
innovative performance aids or conventional documentation. 
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About 393,000 enlisted "personnel were expected to complete 
Initial skill training courses (after 8 percent attrition) and 
165,000 to complete skill progression training (after 5 percent 
attrition) (MMTR FY 1981, p. V-4, V-7 ) . The costs of on-the-job 
training which follows school training are essentially unknown 
(they are included among the costs for Operation and Maintenance, 
which are $62.4 billion in FY 1982 (Brown 1981, p. 312). 

The three Services spent over $5 million in FY 1979 for re- 
search and development on maintenance simulators; this amount is 
projected to decrease to about $1.6 million by FY 1983 (Table 2). 
About $3.7 million (68 percent) of the FY 1979 funds (category 
6.4 funds) were for the development and procurement of prototype 
'equipment; about 49 percent of all funds for FY 1980-1983 would 
also be allocated to prototype equipment. Maintenance simula- 
tors either under contract or planned for development, as of 
February 198^, are listed in Table 3. 

Over a 7-year period (FY 1975-1981), the Naval Training 
Equipment Center alone procured .training equipment at a cost of 
$649 million; planned procurements as of March 1980 were for 
an additional $305 million. Maintenance trainers will account 
for $3.2 million or 0.3 percent of these procurements; equip- 
ment with a unit cost less than $100,000 is not included in 
these figures (private correspondence, NTEC N-7, 8 March 1980). 

The Air Force Air Training Command estimates that the cur- 
rent inventory of all maintenance training devices cost $500 
million, of which $350 million is for aircraft maintenance alone 
(Aeronautical Systems Division, 1978). There are thought now" 
to be- about 3600 different types of maintenance training devices 
in the inventory to support aircraft systems. The procurement 
of maintenance simulators for the F-16 aircraft is estimated 
to cost about $32 million (this includes some units to be 
delivered to NATO countries). 
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TABLE 2. FUNDS FOR R&D ON MAINTENANCE SIMULATORS BY MILITARY SERVICES 

FY 1978 - 1983 
Funds, (thousands of dollars) 





FY 78 












Service/PE 


and 


FY. 79 


FY 80 


FY 81 


FY 82 


FY 83 




Prior 












Array 














6.2 


47 


600 


800 a 


650 a 


600 3 


900 a 


6.4 


2000 


2000 


2000 




-- 


— 


(Total) 


(2047) 


(2600) 


(2800) 


(650) 


(600) 


(900) 


Navy 














6.3 


2723 


362 


516 b . 


1210 b 


lT25 b 




6.4 


1703 


1665 


233 C 


275 C 


302 c 


211 


(Total ) 


(4426) 


(2027) 


(749) 


(1485) 


(1427) 


■ ( 211 ) . 


Air Force 














6.3 


600 


800 


640 d 


800 d 


700 d 


500 d 


(Total) 


(600) 


(800) 


(640) 


(806) 


(700) 


(500) 


TOTAL ' 


7073 


5427 


4189 


2935 
L 


2727 


1611 



PE 62727A-230 BO: AMTESS II, Software, BITE/AMTE. PM TRADE FY 1981 
Apportionment Review, TO June 1980. 

b PE 63733N W 1202-PN IMTS; W 1201-PN IHOMS; W 1207-PN ATE NTEC 
R&D Program, February 1980. 

C PE 64703N W 0784-PN SAMT. NTEC R&D Program, February 1980. 

PE 63751F 2361, 6883, Flat panel simulator. FY 1981 AFHRL 
Apportionment Revi^ Data Book, 27 June 1980. 

Sou-ce: Joint Technical Coordinating Group ■■ Sub-group for Main- 
tenance Simulators, December l978 (Draft), with modifica"- 
tions noted above. 
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TABLE 3 ' MAINTENANCE TRAINING SIMULATORS* UNDER CONTRACT 
FOR PRODUCTION OR DEVELOPMENT OR PLANNED FOR DEVELOPMENT 
(as of February 1981) 



System Simulated or 
Simulator Designation 



AMTESS (Army Maintenance 
Training and Evaluation 
Simulation System) 



EEMT (Electrlcar/Elec- 
tronic Maintenance 
Trainer) 



Fire Control/Search Radar 
Maintenance Trainer 



6E and UH67/_ 
Series Trainers 



Service 



Maintenance 
Echelon 
Trained 0 



RQlAND institutional 
Trainer 



Type of 
Simulator* - 



Device/Program 
Characteristics and Status 



2-0/ 3-D 



2-0/3-0 



2-0 



0 ant) I 



P1RIF1NDER 



0 and 1 



2-D/3-D 



2-D 



To provHe initial training in di- 
verse skill areas. Contract let 
In December of 1980 for delivery, 
in October of 1981 of two "bread- 
board" units for evaluation. 

* Initial (Navy A-school) training 
for electronic and electronic war- 
fare technician ratings. Con- 
tracts awardea in J^e and July 
of 1980 for delivery of twenty 
2-diK*nslonal and two 3-dimension- 
al prototype units. 

Initial (Navy A-school) training 
for fire control technician 
rating. Front-end analysis com- 
pleted. RFP planned for release 
in March with contract award for 
units anticipated by August 1981. 

Small flat panel devices for 
basic skill training (Navy A- 
school) In several skill areas. 
Contract awards to two firms en* 
compass 20 simulations "and 194 
trainers, deliveries on one con- 
tract are scheduled to be com- 
pleted in February 1981 . 
Deliveries on the second contract 
are scheduled to begin In the 
spring of 1981 and to be com- 
pleted In December. 

Training m electronic and hy- 
draulic systems at organization- « 
al and direct support (OS) eche- 
lons. RFP released In December 
1980. contract award planned for 
October 1981. Contract is to 
include five organizational 
trainers, two OS echelon 
trainers, and two mockups. 

Provides operator and mainten- 
ance (organizational and 
intermediate echelons) training 
for mortar- and artillery- 
locating radars. Contract award* 
ed August 1977 for 36 trajners 
for operator training of both 
radars and* raWtenance training 
of the mgrtar-lapftting radar; 
deliveries began In January 
1980 and should be completed in 
early 1981. Maintenance train- 
er for artillery-locating 
radar will be developed/pro- 
cured on a subseouent contract. 
(See Randle 1980). 

(Continued) 
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TABLE 3. (Continued ) 





System Simulated or 
iinxjldior ursiyn#vion 


t 

Service* 


Maintenance 
Eche Ion 
Trained 


i 

Type of 
Simulator i 


Oevlce/^rogram 
Characteristic's and Status 


•! 




REF.S (Reactive Electronic 
Equipment S'">ulator) 


A 


0 




f'lovides operator and organ- 
isational maintenance training 
for tne Army Tactical Co*runi- 
cat'ons System (ATACS). Contact 
awarded In September 1977 for 
one *• station network. Delivery 
currently anv*cipa<ed for old to 
late Summer I981. 


t 

! 
i 
1 




FVS (nghting Vehicle 
System) Maintenance 
,» Tra i ncr 


A 


C and I 


ho 


Contract has ieen let for cesfgn 
and data. Contract for fabric- 
ation planned to be funded 'rom 
FY 1982 budget and to Include 
four different simulations and 
, a hands-on trainer? all for' 
turret maintenance. 


t 
* 

• 




XH»1 Tank Maintenance 
Training, System 


A * 


0 


2-o« 


In procurement; Includes six 
simulations (covering five tan* 
su&systems) and hands-on 
trainer. Delivery 0/ prototypes 
Is scheduled to begin In Feb- 
ruary 1981. Delivery of produc- 
tion versions Is scheduled for 
July 1982 to February 1984. 






♦ 

H109/H0 Turret Trainer 




0 


2-0 


!n procurement, deliveries to 
begin In July 1981. One Sim. 
ulatlon provides training In 
electrical and hydraulic main* 
tenance fjr self-propelled 
artillery. 






1PR (Integrated Radio 
Room) 


% 


0 


3-D 

U 


Provides operator and mainten- 
ance training of Trident sub- 
marine communications system. 
One system (consisting of a 
simulated coemunl cat ions system 
and several pa rt-task^- trainers ) 
-as placed under contract In 
September 1979 and is scheduled 
for delivery in March 1981. » 
second system rna^be procured. 






Ml 92 TtS (Fire Control 
System 


N 

t 


0 




Currently on letter contract (to 
be evade def.nlte in February or 
March 1981], with first deliver- 
ies scheduled for March 1982. 
Configuration Is a modification 
of the T1CCIT system that inte- 
grates conventional flat panels 
in a 12-student-station TJCCP 
complex. Contract will involve 
two complexes. 






CIUS (Phalanx Close-*n 
Weapon Systen) 




0 




tetter contract (signed in Oct- 
ober 19S0) to be made definite 
in February 1981 with deliveries 
scheduled to begin 'in November 
1981 , Contract provisos 
specify delivery of 39 Jets ,0* 

(Continued) 
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TABLE 3. (Continued) 



System Simulated or 
Simulator Designation 


Service 3 


Maintenance 
Echelon 
Tramed b 


Type of 

j imu i d lur 


Device/Progrdn 1 
Charac ten sties and Status ' 










eight simulations v ( pane Is por- 
traying different subsystems) 
and one 3-dlmensiooal model. 


00 963 Waste Heat 8o11er 


N 


0 




Contracted in December 1980 for j 
delivery of one set of three j 
Simulations in December 1981 j 
Firm design will not be set un- | 
til March br April 1981. 


SQQ-89 Sonar Maintenance 
Trainer (formerly the 
Generalized Sonar 
Maintenance Trainer) 

m 


N 






Large program contittmg of op- 1 
(ration and maintenance trainers } 
for three sona r sys terns ■ Com- j 
Plete program planned to consist ! 

. of both simulation and stimulated . 

/operational equipment. RFP for 
procurement of operator trainers j 
scheduled to be^ issued in March ! 
1981, with resulting contract to i 
be funded from FY 1981 budget. 
Maintenance tratner front-end 
studies to be contracted f rom 
FY 1 981 funds with procurement 
planned from FY 1983 budget. 


Magea ABC (Automata 
Boiler Control ) 


N 


0 


2-0/3-D 


Prior contracts resulted in pro- \ 
curement and evaluation of three 
units employed for research. ; 
Current funding is to modify the ' 
three simulators to the current | 
configuration of the operational 
equipment for use in mainstream ( 
training. , 


Woodward governor 

v- 

1 


N 


0 


- 


Research program. Front-end 
analysis essentially completed. » 
Current funding provides for 
design/development of audio-visual 
and courseware (other than EDP) j 
materials. Design/development of 
hardware and EDP software/course- | 
ware to,be initiated with 'uture 
year's fundfng. j 


A6-E TRAM ORS (Dectection 1 
and Ranging System) 

l 

i 


N 1 
1 


0 


3-D 


Two trainers delivered under prior ; 
contracts. Current funding is 
limited to updating these devices 
to the current configuration of 
the operational equipment. 


EA-68 ICAP.I'TJS 
(Tactical Jawing j 




o 


3-0 


Procurement contract awarded in 
December'1980 for t*o units. Deliv- 
eries are scheduled for January and 
March 1982. Current planning in- 
cludes later modification of at 
least one unit to the ICAP-1I air- 
craft configuration. 

(Continued! 
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TABLE 3. (Continued) 





System Simulated or 
Sihwlator Obsignation 


Service 4 


Maintenance 
Echelon. 
Trained 0 


Type of 
Simulator 


Device/Program 
Characteristics and Status 




AT Trainer 


N 


0 


3-D 


Military m-house program intiated 
In Oecember 1975. Delivery of 
partially configured tMlners in 
Junp 1979 (1 device) and December 
198o (2 devices). Modification 
of delivered trainers to full 
design configuration Is scheduled 
for completion In November 1981. 




MA-3 Test 8ench/CSD 
(Constant Speed Drive) 


N 

f 


I 


3-D 


Research program. Single device 
delivered in August 1980. Current 
funding provides support for on- 
site training evaluations. 




SH-3 8lade-fold Trainer 


H 


0 




front-end analysis scheduled for ■ 
completion during Sumner 1981 . No 
decisions regarding the program 
are anticipated before that tine. 




. F-1B Maintenance Trainers 

•> 


H 


* 


-- 


Front-end study recommended a clx 
of simulations ana operational , 
equipment trainers. RFPs for five 
simulators were released in Jan- 
uary 1981. Contract awards are 
expected to begin in April 1981. 
Procurements will be managed by 
McDonnell Douglas as sub-contracts 
to, the basic weapon contract . 


- 


6883 Test 8ench 
(Flat Panel Trainer) 


AF . 


I 


2.D 


Research program. Single device 
delivered in August 1980. FY 
1981 funding Is to Support an on- 
site training evaluation. 


* 


r • 16 Maintenance 
Simulators 


Ar 


0 

- 


2.D/3-0 


Initial contract (September 1977) 
provided for delivery of six sets 
of 18 simulations. Deliveries of 
six sets of 12 modified (degraded) 
simulations were completed by 
September 198D. FY 1981 and 
later funding is to provide for 
retrofit of delivered articles to 
their Initial design coofigura- 
tion and production/delivery o* 
the remaining simulations. 




E.JA (AWACS) Radar 


AF 


0 


2.D 


In development. Contract award- 
ed in September 1980 for one 
simulator, containing 10 student 
stations. Delivery anticipated 
in May 1982. 


A 


E.3A (AWACS) Oata 
Display /Control System 


l * AF 


0 




Front-end analysis nearing com- 
pletion RFP scheduled for 
release in March 1981. Contract 
award anticipated at the end of 
FY 1981 . 

s 

(Continued) 
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TABLE 3. (Continued) 



System Simulated or 
Simulator Designation 



Servce 



Maintenance 
Echelon. 
Trained 0 



Type of 
Simulator 



Device/Program 
Characteristics and-Status 



1 



EA-6B iCAP-M CNR 
(Communication/ 
Navigation) 



E-3A (AWACS) Advanced 
Radar Maintenance 
Trainer 



AN/TPS-43E Radar 



AF 



3-0 



Front-end analysis and detailed 
specification, scheduled for com- 
plete in April 1981. Funding 
for procurement anticipated from 
the FY 1982 budget. 

For training to advanced Skill 
level . Functional requirements 
study m initial stages. Con- 
tract award not anticipated 
before the end of FY 1982. 

Military in-nouse project. Pro- 
gram initiated 1n early 1977. 
First device placed in use in 
late 1978. A second unit 1s 
currently being fabricated 



a p 
A, Army, N: Navy, AF- Air Force 

°I. Intermediate-level maintenance, 0 Oragnizat'onal-level maintenance. 
c Two-d1trens1onal , three-dimensional. 

Proposed for M 109/ 1 10 howitzer turret, M60 tank, M809 truck, rdddr illuminator. 
e Also called the* Simulated Avionics Maintenance Trainer. 
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One large industrial contractor has estimated that the 
Department of Defense will spend about $620 million for mainte- 
nance trainers over the period 1977-1985; annual procurements 
are estimated to reach about $120 million per year by 1985 
(Fig. 1). The_ distribution of thi^ j^qcurement, accord i .g to 
type of trainer, is predicted to be as shown in Fig. 2. Out- 
side the United States, the procurement of maintenance simu- 
lators is estimated to be about $5.5 million per year. 




YEAR 



FIGURE 1. Estimated procurement of maintenance 

trainers by the* Department of Defense, 
1 97 b - 1 98 5 (as of November 1 979 ) 
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IZtMt 

FIGURE 2. Predicted procurement of maintenance trainers 
by the Department of Defense, according to 
type of application, 1977-1985 (estimate 
made in November 1979) 

The "Electronics-X" study, conducted in 1974 , was a ma.jor 
effort to determine the cost and reliability of military elec- 
tronic equipment (Gates, Gourary, Deitchman, et al., 1974). 
Four methods were used to estimate the cost of maintaining 
electronics equipment each year. The results ranged from $3.4 
billion to $6.8 billion, with an average -of $5.4 billion per 
year (Gates, Gourary, Deitchman et al., 1974, Vol. II, p. 374). 
The estimate of $5.4 billion per year for maintenance is about 
equal to the cost of procuring electronic equipment each year 
(Gates, Gourary, Deitchman et al., 1974, Vol. I, p. 52). Note 
that procurement costs relate to acquiring current technology; 



the maintenance costs relate to systems whoso average age is 
about 10 years* 

Advanced military equipment has become more complex in order 
to provide improved performance* Increased complexity brought 
increased cost and decreased reliability, the latter imposing 
increased demands on maintenance personnel and^pxsour'ces ♦ The 
Electronics-X study showed that the reliability^ avionics 
equipment in the field decreases with increases in unit cost 
for aircraft in accordance with the following relationship: 

Aircraft MFHBF* = 1.3 x 10 6 /cost . 

As shown in Figure 3, more expensive (and more complex) elec- 
tronics equipment has a lower reliability and creates a larger 
demand on maintenance activities than does less expensive 
equipment • (Gates , Gourary, Deitchman, et al. 1974, Vol. I,. p. 
56). A similar relationship, based on limited data, was foCind 
for Army Area Communications Systems (AACS) where 

AACS MTBF** = 10 7 /cost ♦ 

The costs for manpower were estimated by a Defense Science 
Board (DSB) Task Force on Electronics Management to account 
for perhaps as much as 75 percent of! the military electronics 
maintenance costs; actual costs are unknown-due to limitations 
in the cost allocation system (DSB, 1974, p. 14). 



*Mean flight hours between failures, 
**Mean time between failures* 
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• R - SPECIAL RELIABILITY PROGRAM 

▲ WF = AF RELO MTBF EXPERIENCE 

WITH WARRANTEO EQUIPMENT 
(CAROUSEL IV) J 

■ WL - AIRLINES RELO MTBF EXPERIENCE 
WITH WARRANTEO _j 




100 200 500 1000 2000 5000 20,000 100.000 500,000 2,000,000 

10,000 50,000 200,000 1,000,000 

UNIT PRODUCTION COST (dollars) 



FIGURE 3. Avionics field reliability versus 
unit production cost 



2 4 

. 33 



D. TYPES OF MAINTENANCE SIMULATORS 



Maintenance simulators now under development differ notably 
in -their resemblance to actual equipment, their functional capa- 
bilities as instructional devices, and in their complexity and 
cost. Modern maintenance simulators are often characterized 
as 2-D or 3-D devices, i.e., as being two- or three-dimensional 
in their physical form; some simulators contain both 2-D and* 
3-D components. 

The 2-D devices consist of flat panels with drawings of 
major components connected symbolically by flow diagrams to 
show electrical and/or hydraulic functional connections be- 
tween components. The panels contain functioning instruments, 
signal lights, and controls, so that the technician can turn on 
power to the equipment, see if it is working correctly, and 
observe the effects of various actions he may take to identify 
and correct the malfunctions that are present. Such panels 
perform as if they were real equipment because each contains 
a computer, with a mathematical model of the real system that 
makes the displays respond appropriately to all settings of 
the controls under all environmental conditions likely to be 
encountered. By setting a switch on his panel, the instructor 
can select a malfunction from a large set contained in the 
computer. The equipment scores the student's performance and 
tells him whether he has correctly identified a malfunction. 
The instructor can s op the sequence of activities for instruc- 
tional purposes, to repeat what the student has done, and 
demonstrate the correct way of isolating a malfunction; this 
is done automatically in some simulators. 

The manufacturers of 2-D simulators have developed soft- 
ware packages and computer and support equipment that can be 
.used with a number of different panels. This has led us to 
distinguish between what later in discussing costs we call 
"standard" and "non-standard" maintenance simulator systems. 



25 



. Standard systems, whether they are 2-D or 3-D simulators, are 
likely to cost less. than non-standard systems. 

A 3-D maintenance simulator looks and perform* very much 
like the actual equipment it mimics, if it is a tlst bench, 
it will be possible to connect components for calibration, 
checkout, agists needed to identify malfunctions. it will ' 
differ from actuW equipment in that it will be ruggedized to 
withstand student abuse and to prevent exposing students to 
dangerous electrical currents or hydraulic pressures. The 
simulator may not contain all the components present in the 
actual equipment, particularly those- that are not relevant to 
its maintenance; if the equipment contains many identical ^m- - 
ponents, only some will be represented. Thesjr components may 
be precise physical copies; in sojne -easel*; they are only accu- 
rate photographs (etcjbted-tjri plastic or metal) with active test- 
points for- making" test measurements. Being under computer con- 
-tr-ol. all components perform or respond as. if they were actual 
equipment; components may be tested, removed, and replaced. 
A 3-D simulator permits "hands on" practice- in the manual 
maintenance skills not possible on most 2-D simulators; it 
also has greater physical similarity to the actual equipment. 
Whether or not greater physical similarity increases the 
effectiveness of training is not considered in this discussion. 

K. OTHER INFLUENCES ON MAINTENANCE 

Many factors beyond training and the use of actual or simu- 
lated equipment can profoundly influence our ability to maintain 
military equipment. These are noted here but they extend far 
' beyond the scope of this paper. Among these factors are the 
quality of personnel recruited by the military Services (and 
thereby available for training as maintenance technicians), 
policies used by the Services to assign recruits to various 
occupational specialties (thereby influencing the quality of 
personnel who become maintenance technicians), the amount and 
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type of training to be accomplished at technical schools (as 
distinct from that to be accomplished on the job), and the 
complexity of the information that must be acquired in order 
to accomplish maintenance* Some factors that influence mainte- 
nance have little to do with personnel and training; these. - 
include equipment design and maintenance poJLLcy. y Vhe design 
of equipment influences both Jtfcfi. nee"8" for maintenance (mean 
time between £aHyce4 r ~and the means for accomplishing it 
whenever- required (e.g., ease of access to components, built- 
in test points, manual or automatic fault detection). Maint- 
enance policy determines whether failed components should be 
repaired or replaced, the availability of spare parts, tools, 
test equipment and up-to-date technical documentation. 

F. ADVANTAGES OF MAINTENANCE SIMULATORS 

The advantages of simulators for training maintenance 
personnel have been argued for more than 25 years (e7g., R.B. 
Miller i Q 54, Gagne 1962, f.umsdaine 1960, Valverde 1968, Kinkade 
and Wheaton 197 2, G.G. Miller 1974, Montemerlo 1977, and Fink 
and Shriver 1978 ). The uvi-jor advantage of a maintenance simu- 
lator is that, as a training device, it can be designed to pro- 
vide facilities important for instructing students, in contrast 
to actual equipment that is designed to operate effectively in 
an operational environment. 

Maintenance simulators can be designed to include, a large 
variety of faults with which maintenance personnel should be 
familiar, including faults that cannot be demonstrated con- 
veniently on actual equipment trainers or that occur rarely 
in real life. All modern maintenance simulators incorporate 
some type of computer Fupport. Thus, the symptoms of many 
types of complex faults can be stored in the computer and se- 
lected simply by a control setting on the instructor's console. 
Computer-supported equipment can also record what the student 
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does, thereby,. cedaCt rig- the- need for constant observation by the 

--instructor. The instructor can use information collected by 
• - 

the computer- to guide each, stydervt; a -computer can also assist 
the student without an instructor's intervention. Records of 
student, performance and achievement cap be maintained automati- 
cally. Simulators can be made rugged enough to sustain damage 
or abuse by students and thus provide greater reliability and 
availability in the classroom than is often possible with actual 
equipment. Training which would be avoided because of safety 
reasons, e.g., -exposure of students to dangerous electrical 
charges or hydraulic pressures, can be undertak n with little 
risk with a simulator. If students using such equipment com- 
plete their training in less time, as has often been the case 
with computer-based methods of instruction, thfere are potential 
cost benefits due to savings in student time, increased- student 
throughput, and reduced need for instructors ermf support per- 
sonnel. 

As noted above, a simulator need not contain all the com- 
ponents found in the actual equipment. Thus, it is often pos- 
sible to build a simulator that offers greater flexibility and 
capacity for training at a "cost less than that for an actual 
equipment trainer. , 

G. DISADVANTAGES OF MAINTENANCE "SIMULATORS 

There are also some disadvantages to the use of simulators. 
The procurement of maintenance simulators necessarily involves 
costs to design and build this special equipment, and to develop 
course materials, maintenance procedures, and documentation. 
The types of training provided by simulators may not provide 
the student with all the skills needed to maintain operational 
equipment, an outcome that seems assured when actual equipment 
is n^ed for training. A simulator may not be ready when needed 
for training because its design and development requires some 
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effort in addition to or at loast parallel to that needed for 
the actual equipment (which is being produced as part of some" 
system); modifications in the design of the actual equipment 
may delay completion of the simulator, if it also must be modi- 
fied. If 'there are many and frequent mod i f icat ions , ' the orig- 
inal simulator may not resemble the operational equipment 
closely enough to be useful for training. / 

Data on the ef f ectTv^rre^is and cost of maintenance simula- 
tors and actual equipment trainec^are considered in the follcw- 
ing chapters. \ 
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II. THE EFFECTIVENESS OF MAINTENANCE SIMULATORS 

\ The purpose of maintenance training, whether with simulators 
o$ actual equipment, is to qualify technicians to maintain equip- 
ment in the field. In fact, however, the effectiveness of main- 
tenance simulators for traininq technicians has" been compared 
to that of actual equipment only on the basis of student perform- 
ance at school and not on the job; there is one exception to this 
qeneral statement (Cicchinel 1 i , Harmon, Keller, et ah, 1980). 
The lack of job performance data to validate traininq (and 
other activities relevant to personnel, such as recruitment, 
selection, and reimbursement) ^appl ies qonerally to all types of 
military traininq and not only maintenance traininq. 

A, EFFECTIVENESS OF MAINTENANCE SIMULATORS AT SCHOOLS 

We found 19 studies, conducted over the period of 1967 to 
1980, that compare the effectiveness of maintenance simulators 
and actual 'equipment trainers for traininq in a variety of 
courses at military traininq schools; these are described in 
Appendix A, Only 12 of these studies provide enouqh detailed 
information to permit meaninqful comparisons; those are sum- 
marized in Table 4. 

Relatively complete data were- found on five maintenance 
simulators evaluated in 14 different courses, e.q., radar, pro- 
pellers, enqines, fliqht controls, FM tuner, test equipment, 
and the Haqen Automatic Boiler Control; most are associated 
with aviation. Ti>ese courses varied in lenqth from 3 hours to 
5 weeks (median 4.7 days, N - 12 courses); the number of subjects 
trained with simulators in these courses varied from 6 to 56 
(median 16, N = 14 qroups)'; a qrand total of 267 students was 
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involved in all of these studies.) Ff f ect i veness was evaluated 
by cbmpannq the scores' of students who used simulators with 
those of students who ur,ed actual equipment trainers in end-of- 
course tests. There are 13 comparisons; in 12 of these, students 
teamed with simulators achieved test scores the same as or 
better than those trained with actual equipment; in one case, 
scores were lower. The differences, thouqh statistically siq- 
nif icant , have little practical sdqni f icance . 

s? 

Cicchineili, Harmon, Keller, et al., 1980,' compared super- • 
visors* ratinqs of on-the-"job performance of technicians trained 
either with a maintenance simulator (the 6883 Test Station 3-D 
Simulator) or actual equipment trainer. Twq field purveys pro- 
vided data on the job performance of 85 and 56 qraduates, 
respectively (some twice); these comprised 7 4 and 49 percent, 
respectively, of the students in the onqinal sample at school; 
some course qfaduates were on the -job for periods of up to 32 
weeks. The superv i sors' d id not know how the students had been 
trained. Their ratinqs showed no noticeable difference between 
the performance of technicians trained with the simulator or 
aetual equipment trainer. The abilities of the technicians . 
i ncreased with amount of t ime on the iob. 

>ffTe automated and individualized method of instruction that 
is an inherent characteristic of modern maintenance simulators 
should be o*pected f to save-some of the time students need to 
complete the same course when qiven by conventional instruction 
(Orlansky and Strinq ]979). Such time savinqs are reported- in 
three of these studies (Parker' and DePauli 1967; Riqney, Towne, 
Kinq, et a I.- 1978 ; -and Swozey 1978 ); compared to the use of 
actual equipment trainers, maintenance simulators were found in 
these studies to have saved 22, 50, and 50 ^percent, respectively, 
of 4 the time students needed to complete the course^. Al thouqh 
no explanations are offered for these time savinqs, on< could 
surmise that they are duo to factors such-^t£, the fact that 
briqhter students can complete a self-paced course faster than 
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one qiven by conventional, q roup- paced instruction, that main- 
tenance simulators generally have cjreater reliability tn the 
classroom than do act ual . equ i pmon t trainers, and that instruc- 
tors need less time to set up traininq problems and/or to insert 
malfunctions in simulators than in actual equipment trainers* 

Based on questionnaires administered at the completion of 
the courses, students favor the use of simulators in 9 of 10 
cases and are neutral in one* Instructors are equally divided 
(about one-third in each cateqory of response) in beinq favor- 
able, unfavorable, or neutral in their attitude toward the use 
of simulators. 

Overall, maintenance simulators appear to be as effective 
as actual equipment trainers for traininq military personnel at 
schools; there is only one contrary findinq. Some of the pre- 
sumed advantaqes of simulators were not examined in these studies 
and therefore cannot be evaluated, e.q., their ability to teach 
students how to correct a wider variety of malfunctions than 
can be done with actual equipment, their superior availability 
compared to actual equipment trainers, and their ability to meas- 
ure and report student performance both to students and instruc- 
tors. The findinqs do not suqqest ways in which the use of 
maintenance simulators could be ^hflproved or where their use is 
likely to be more effective* There arc no cases, except for 
Cicchfnclli, Harmon, Keller, et al. 1980, where the effect of 
traininq upon iob performance is examined; they found no differ- 
ence between a simulator and an actual equipment trainer; how- 
ever, Cicchinelli, Harmon, Keller, et al. do not report the 
amount of transfer of traininq from school to the idto, i.e., 
transfer effectiveness ratios. * 

B . R KhEVANT DATA FROM COMPUTER- RASH) INSTRUCTION 

Modern ma i ntonance s l mu 1 a tor s can prov lde l ndi vidua 1 \ ?od 
instruction on a series of prescribed lessons.. They can also 
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measure student performance* and see that the student does not 
qo to a now lesson until he has mastered the preceding ones. 
The instructional strateqies employed in these simulators are 
derived from wideiy used methods of instruction called computer- 
assisted and compute t-manaqed instruction; both are ind i vidua 1- 

9 

ized and self-paced in nature and use computers to monitor 
s t uden t proq ress . In computer-ass i s ted instruction ( CAI ) , all 
the instructional material is stored in a computer and presented 
to the Relent in a controlled manner, e.q., via a cathode ray 
tube or a visual projection device with random access to a larqe 
reservoir of slid.es. The student responds to this material by 
touching portions of the screen sensitive to touch or by using 
a keyboard or teletypewriter. In computer-manaqed instruction 
(CMl), the lessons are performed away from the computer in a 
learninq carrel or on a laboratory bench set-up. The student 
takes a test at t;he completion of each lesson; the answers, on 
a sheet, are scored by the computer* which then directs the stu- 
dent to a new lesson or to additional practice on the current 
one. 

CAI and CM I systems are not maintenance simulators but 
they have been used to provide certain aspects of maintenance , 
training, "e.q., knowledge of operating pr i nc i pies , ,t rouble- 
shoot inq procedures, fauft identification, and the knowledge 
aspects of remove and replace act ions (i.e., what the tochn lcia'n 
should do after a fault io identified rather than replace 
actual parts). Knowledqe about maintenance procedures can be 
acquired' on a CAI and CMI system, but this is accomplished with 
less fidelity and with little of the hands-on experience that 
can be provided by a maintenance simulator, particularly of 
the 3-0 variety. Pi sewn ere in this paper , where we cons ide r 
costs, we characterize some maintenance simulators as CAI-like. 

In a previous study, the authors examined the cost- 
< f f ect i v^ncss of computer-based instruction in military training 
(Orlansky and String, 1979). Somo of the courses on which 
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effectiveness data were available involved instruction similar 
to that provided on maintenance simulators,' i.e., basic elec- 
tronics, vehicle repair, fire control system maintenance, pre- 
cision measurinq equipment, and weapons mechanics. Data on 
student achievement in these courses are presented in Table 5; 
there are 28 data points which compare conventional instruction 
to the use of CAItand two to CMI . Student achievement in these 
courses at school with CAI or CMI was the same as or superior 
to that provided by conventional instruction; the amount of 
superior performance, when present, had little practical siqni- 
f icanc£ . 

Data on ,the amount of student time saved by CAI or CMI in 
these courses, compared to conventional instruction, are shown 
in Table 6; there are 30 data points. The amount of time saved 
by computer-based instruction varied from -32 to 59 percent, 
with a median value of 28 percent. 

These data on student achievement and on student time 
savinqs with computer-based instruction are consistent with 
that reported above for maintenance simulators. Orlansky and 
Strinq (1979) found that students favor computer-based instruc- 
tion while instructors do not. They also found that . computer- 
based instruction may increase student attrition, a matter not 
considered so far in any study of maintenance simulation. 

In summary, the data show that maintenance simulators are 
as effective as actual equipment when used for traininq mili- 
tary technicians. These results are consistent with the results 
of studies of computer-assisted and compu te r-manaaed instruction 
in courses that provide technical information similar to that 
provided in maintenance traininq. A few studies show that 
maintenance simulators save student time hut most studies did 
not address this issue. Students favor the use of maintenance 
simulators; instructors favor, are nelitral about, or do not 
fc favor such simulators in about equal numbers. 




TABLE 5 STUDENT ACHIEVEMENT AT SCHOOL IN COURSES RELEVANT TO MAINTENANCE , 
CAI AND CMI COMPARED TO CONVENTIONAL INSTRUCTION 
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•TABLE 6. 



AMOUNT OF STUDENT TIME SAVED IN COURSES RELEVANT TO MAINTENANCE, 
CAI AND CMI COMPARED TO CONVENTIONAL INSTRUCTION 
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Claims'have been made that maintenance simulators are 
superior to actual equipment for training because of their 
capability to demonstrate more malfunctions, provide qreater 
freedom from br£ajidown in the classroom environment, provide 
an opportunity to sav£' instructor time, and so on* No studies 
were found that examine these capabilities. No data were found 
on student attriTion when simulators are used* 

C. PERFORMANCE OF TECHNICIANS IN THE FIELD 

The effectiveness of maintenance training i % s determined 
ultimately by how well maintenance personnel* perform in the 
field rather than at school. Only Cicchinelli, Harmon, Keller, 
et al. 1980, among the studies we were able to find, compared 
the performance of students trained with a simulator (the 6883 
Test Station 3-D Simulator) or actual equipment trainer for 
varying periods ,-of time after leaving school. According to 
ratings made by supervisors, no differences were found between 
both qroups of students. 

The military services use five large data management 
systems to provide detailed information on the current main- 
tenance statOs'of military equipment. These data systems 
are' identified below: * 

Service Maintenance Management System '_ 

Army TAMMS The Army Maintenance Management 

System 

Navy > Ships' 3-M The Naval Ships 1 Maintenance and 

Material Management System 

Navy Aviation 3-M ' Naual Aviation Maintenance and 

x . Material Management System 

Air Force 66-1 and 66-5 Air Force Maintenance Management 

Systems 
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We examined the possibility of using data available in these 
systems to describe the performance of maintenance technicians 
in the field (see String and Orlansky, 1981)* If this yielded 
useful information, we might be able to compare, for example, 
the real short- and long-term effects of training personnel 
with maintenance simulators or with actual equipment trainers ♦ 
We know, as was shown earlier in the chapte'r, that both are ^ 
about equally effective at school. 

As presently constituted, these systems cannot provide 
information useful for assessing the effectiveness of alter- ' 
native methods of training. In a more general sense, this 
applies also to information needed to validate many personnel 
practices, such as recruiting, selection, and policy on pay and 
allowances. The names of individuals who performed maintenance 
actions are not kept in the records maintained in the central 
data files. The ability to identify and track individuals is 
a mandatory requirement in any attempt to relate method of 
training with subsequent performance. This type of data is 
kept only at the field activities but it is discarded after 
6 months. The use of maintenance records with personal iden- 
tification for analytical purposes would require special methods 
of processing in order not to infringe on provisions of the 
Privacy Act. Even so, such, records are not precise enough to 
distinguish what parts of a maintenance action were performed 
by a particular individual, particularly when the work is 
performed over more than one shift. The practice of cross-skill 
maintenance, to train individuals to maintain a wide variety 
of equipment under combat conditions, assigns individuals to 
tasks for which they were not trained at school and it would 
complicate any analytical effort. in brief, it was concluded 
that presently available maintenance data records can not be 
us6d to assess the effectiveness on the job of various methods 
of training at school; it is conceivable that these systems 
could be modified to provide the data that would be needed. 
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That the performance of maintenance technicians affects 

the quality of maintenance can hardly be doubted. A few studies 

have examined this possibility by analyzing selected data on 

components removed for replacement or repair that were found 

later not to contain any malfunction. These studies examine 
« 

data produced by a group of technicians in a particular work 
center; they do not review the performance of individual techni- 
cians and do not address the method('s) by which these technicians 
were trained. * 

Findings from seven studies are summarized in Table 7. 
All involve corrective maintenance at the organizational level, 
although one also involved intermediate maintenance. Most 
concern maintenance of aircraft, a few of surface vehicles. 
The periods of observation are relatively long (6 months or i 
year? one is for only one month). The removal of non-faulty 
parts, in these studies, accounted for 4 to 43 percent of all 
corrective maintenance actions and 9 to 32 percent of all 
maintenance man-hours. One study (Gold, Kleine, Fuchs, et al. , 
1980) found instances where faulty parts were not removed and 
where good parts were damaged during corrective maintenance. 

These findings suggest strongly that, properly modified, the 
maintenance data systems might provide data on human performance 
useful for validating different methods of training. even so, 

? it is well to recognize that not all instances of removal of 
good parts necessarily imply inadequate performance of tech- 
nicians. Such removals could also be due to inadequate test 
equipment that cannot distinguish between good and bad parts. 
It is also possible that, when under great pressure to return 
equipment to an operational status, technicians may deliberately 
remove and replace a large number of components just t;o make 
sure that the faulty ones have been eliminated. validation 

✓ of training devices and procedures would probably need more 
data on job performance than -just that concerning the unneces- 
sary removal of good parts. 
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TABLE 7. SUMMARY OF STUDIES OF ORGANIZATIONAL ECHELON CORRECTIVE MAINTENANCE 
■WHERE NON-FAULTY PARTS WERE REMOVED 
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in. * costs qf Maintenance simulators 

A. INTRODUCTION 

This, chapter discusses the costs of maintenance training 
simulators. Three classes of simulators are defined, their 
characteristics and uses within the Services are discussed, and 
a structure of data for analyses of their costs is formulated. 
Available cost data for maintenance of simulators are discussed 
in terms of the problems, with respect to costs, that arise 
from the^r physical characteristics, procurement quantities, 
and contracting practices. The costs and characteristics of 
selected simulator programs are presented in Appe- .x B. 



3. CLASSED OF SIMULATORS 

• With respect to the costs of maintenance training simula- 
tors, it is useful to distinquish among three classes %t de- 
vices, denoted here as "standard" systems, "non-standard" sys- 
tems, and "CAI-like M systems. Differences amonq these three 
f types lie in the following areas: 

• Physical characteristics, 

• Complexity and cost, 

• Extent of use within th$ Services (i.e., the inventories 
of devices in use and under contract), and 

• Contracting practices employed in^their procurement (and 
hence cost data that are available). ^ 
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i. Standard Systems 

The critical^distinction between standard and other classes 
Of maintenance simulators is standardization of the physical 
configuration. Simulators of this class consist of two elements 
one element, called here the HM general simulation system" consti- 
tutes 1 ^ generalized and adaptable (but incomplete) simulation 
capability *th?t can satisfy a wide range of specific traininq . 
applicatiohs. The second element, that tailors the qerioral'" 
simulation system to a particular training application, *is typi- 
cally-limited to courseware and pictorial or other representa- 
tions (i.e., the simulation model) of tfte particular equipment * 
being simulated. Standard systems were the earliest type to he 
used for maintenance traininq and are the only, class to achieve 
extensive use. The three Services have procured close to 650 
simulators for nearly. 200 .separate training appl icat ions 4 train- 

0 

ing courses or course seqments) . x 

£&ly^oiir^xoju^ ^s4*nda*d-HRa4**t-enanoe- 
simulator systems: Educational Computer Corporation (RCC); 1 
Burtefc, Inc.; Rvdqeway Electronics, Inc.; and Lockheed Aircraft; 
Services Co. (LAS). For all but Kidgeway, "this type of simula- 
tor is only one of several product lines; and for all but LAS, 
'these companiesx manufacture onl y . educat iona 1 and tr J aininqequip 

ments. " _ 

bompared with tht? other classes of simulators, the stand- 
ard systems are generally low in cost and limited in terms of 
the complex ity of processes that can be simulated. Development 
of particular traininq applications typically, involves small 
risks. With tow exceptions, these devices have been procured 
^lrouqh fixed-price contracts. 




2. Cost rmpact of Standardizati on 

The four manufacturers have produced six standard simu- 
lator -systems or models. The elements that are typically com- 
mon to a model consist of data-processing hardware (a .ceijtral 
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processor and a partial set of , input/output devices), the soft- 
ware operating,, system, audio-visual devices, a,nd structures for 
Housing all the components of the simulator. Taken together, 
these are generally referred to as the "mainframe" or "console" 

^he components that^ are tailored :tp the particular application 
consist of other input/output devices (typically a display 
panel depicting the operational and test equipment beinq simu- 
lated) and courseware ^in the form of an application program N 

, Icontained in magnetic tape, disc, or plug-i<n proqrammable-read 
only-memory (.PROMJt units]. 

The' size and* structure of display media may- vary within a 
single model, a-nd the same simulation application may be pro- 
duced with two sizes of display panels—one for classroom 
demonstrations and one for individual^ uoe. Advances in micro- 
processor .technology appear to have fostered further variations 
within a model while ret ^n£ng\the essential attributes of 

"STaraaraTza^ ECC model has been delivered with pro- 

cessor memories ranging between 16 and 4 8 thousand bytes. The 
variation in memory size haj> permitted corresponding variations 
in complexity of simulation and the juse of audio/visual~devices, 
For example, the 48-thousand-by te devices procured through an 
Army contract for XM-i tank training will drive a cathode ray 
tube (CRT), printer, random-access slide projector, and an 
audio device in addition to the normal simulator display 
parfel. A contemporary Navy contract (for entry-level skill 
training) specifies the sane model with a 16-thousand-by te 
memory and with" only the display panel. 

The physical arrangement of standard systems appears 
especially adaptable for 2-d imensional trainers. However, " 
3-dimensiqnal simulation is possible. 

One impact ^of standard izat ion .is .interchanqeabi 1 i ty , and - 
this serves to reduc£ costs of both manufacture and repair. 
Individual consol6s may be easily modified to different train* 
ing' applications. Two of the' standardized models are designed- 
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so that the tailored, specific components of any training 
application (the display panel and courseware) can be mated 
with a single console in the classroom or laboratory; a chird * , 
model can be ordered with either " classroom-changeable or fixed 
panels. As a result, a single coiYsble may be used in a number 
of training applications at; the same location 0 . One of these 
'models (the ECU) provides the bulk of the simulutor.s used in 
Naval aviation weapon-specific training. Naval aviation train- 
ing is organize 4 so that both maintenance and a part of pilot 
ground training for one model of aircraft are conducted at the 
same Naval Air Station (NAJS) by Naval Air Maintenance Training 
Detachments (NAMTD) . While a large number of different dis- 
play panels (up to 25J„may be employed for maintenance training 
by one detachment, <* NAMTD will generally have no more than two 
main frames that will be shared by all pilot and maintenance 
training 'courses, 

The more important cost impact of standardization lies in 
the commonality of system software. Available evidence points " 
'to the programming and programming design effort as the major 
cost of non-standard simulator development. * This high cost 
provides a strong incentive for producers to develop a single 
basic software system that is both comprehensive and adaptable 
to a wide range of potential training applications. Develop- 
ment of-such a software system -reduces the programming asso- 
ciated with a particular, training application to a relatively . 
small set of courseware written in a high-level and relatively 
simple language^tihat may (in the case of maintenance simula- 
tion) reduce to a sequential coding of the maintenance proce- 
. dures/steps set out in technical orders. \ 

Commonality of software is- the distinguishing aspect of 
standardization, and 'manufacturers -have placed a heavy emphasis 
on developing versatile software packages. .Once developed, they 
are tightly held, considered proprietary, 'and may (at least in 
part) be hardwired into simulators. In addition, the software* 



packages have" been retained while'other features of the general 
* simulation" systems have been allowed to change. ECC^nas pro- 
duced two generations of standard training simulators; the 
later one employs an advanced, higher capability processor and 
has a quite different physical appearartce, but uses the earlier 
software package. * LAS produces two models of simulators m that 
employ \different types of display panels. Yet, the two employ 
the same software system atfd seem best considered as a single 
general i^ed' system.' * 

• . • \ - 

3. Non-Standard Systems 

The nonstandard systems preserit a picture that is quite 
different from the standard systems, ^evertteen^non-standard 
programs (discussed in Section C f below.) have been initiated; 
with one exception, each appears to involve a complete (i.e., 
groUnd^Tfip) development effbrt. * Taken as "a group, their out-r 
standing characteristic is diversity, encompassing different ^ ' 
contractors and types of contracts, program purpose, numbers 
of 1 devices manufactured, physical characteristics, complexity, 
and cost. j 

Two programs (the AT Trainer and An/tSP-43E radar) 'have 
been in-house projects at mil-itary installations while the 
remaining 15 have been contracted to one or more firms. The . 
15 contracted programs have ijwdlved 10 firms *as principal con- 
tractor; one company (Honeywell) has played .this role on five 
projects. Only one firm (ECC) has also' had experience in pro- 
ducing a standard device; four firms (Grumman, RCA, Hughes, and 
Sperry) also produced the tactical equipment being simulated. 

Three" programs (the MA-3, A-7 HUD, and 6883 -Test 'Benches) * 
have research in maintenance simulation as their principal . pur- 
pose and employ cost-plus-fixed-fee (CPFF) contracts. The othe 
14 programs 4 (including the two in-h use programs) serve main- 
line trainihg. Ten of the 12 that were contracted were funded 



through either x f ixed-pr ice-inpentiye-f ee tFPIF)' or firm-fixed- 
price (FPP) contracts; the , rente in ing twp JMK 92 FCS and Fire- 
fjLn&er) employed ^ost-plus contracting. 

: When,,compl£ted (as currently planned), the 17 programs 
will result,- in the development of 47 unique simulations and 
a 4 el'ivery.*of 6a7<.units, i.e., trainers. ° The MV92 FCS , CIWS, 
and F^16 programs will be responsible for 30 of the different- 
simulations and 632 of the .trainer sY both the CIWS and F-16* 
address' training in a number of skills for a s^ipglfe weapon sys- 

\tein and will result in the development of a family of devices 
with extensive commonality, rather similar to the standard 
systems. • ' Typically, the other programs are. Concerned j^itth 
single, training applicatibns and a single training device. ^ 

The physical characteristics of the non-st&ndard simula- 
tors appear to be similarly diverse. There are two- and three- 
dimensional trainers. Since software is normally closely 
held by contractors, wide variability can be, Expected. Fur- 
ther, since a non-standard system typically simulates only , 
onev tactical system, Ht is not necessary to provide a defini- 
tive separation between software and courseware functions. * 

m he total program costs of the non-standard systems 
(adjusted to current price level.s) differ by factors of up v 
to~390:l,' and the°average costs of devices differ by factors 
of up**to 40: 1. 

4. CAI-Like Systems , . 

A CAI-like maintenance simulator is o computer-assisted in 
struction (CAI) systepi with courseware designed specifically to 

"train maintenance skills. A typical CAI system uses a 2-dimen- 
sional display (CRT* and /or random access slide or micrpfiche 
projector)* to present lesson materials., (p icttires of equipment 

. £nd the like) under control of a computer that also monitors t 
student progress, prescribes* lessons , and scores tests. When 
adapted to maintenance training, the CAI features are retained, 
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• and the trainer may also employ 3-d imensiona^ depictrons of " 
« equipment.. , , ' '• 

.' ' . One experimental system of this -class (the Rigney Trainer) 

f ■ has been built/ and two other systems have recently been placed 
. . : under contracts A contract for , the design and fabrication of 
* ^ • prototype units of the Electronic Equipment Maintenance .Trainer 
' or (EEMT) was 'awarded to Cubic Corporation by the Navy Personnel 
Research and Development'Ceriter (NPRDC) in August of 1980, and 
a„ preliminary design has been formulated. The Army has let, 
several contracts for the study of design concepts for the Army 
Maintenance Training arid. Evaluation Simulation System (AMTESS)' 
and let a contract to Grumman .Aerospace in December 1980 for ' 
. construction of two "breadboard" units for further evaluation. 

• , • ' EEMT is . intended for initial 'skill ( "A-school" ^training., 

primarily in electronics. It is to provide both 2-d imensional 
, displays (generated through a cathode, ray tube) and 3-dimen- 
, sional simulations and is to be capable of simulating a variety 

of particular electronic .systems. This lat°ter capability is the 
basis for distinguishing CAI-like from the other classes pf main- 
tenance simulators. The software system must be comprehensive, 
and adaptable (as in the case of standard systems). In addi- 
? j -tion^both the software system a'nd ( the courseware must Be more 

_ extensive since they m'ust aiso provide the information that 
•would be contained on the display panels of simulators that are 
tailored to a particular training application (whether standard, 
or nonstandard types). 

% The only information available to this project on the costs 
of CAI-like systems is contained in the cos.t proposal for the 
EEMT system., m this proposal, requirement's for labor (of^all 
types) were stated -sin terms of hours, with insufficient infor- 
mation to convert them to dollar cost* to develop an estimate 
■ of total program costs. As a result, ^he CAI-j*Hc^ systems are 
not discussed further in this chapter;- 
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C. SERVICE USE OF MAINTENANCE SIMULATORS - 1 

Service inventories of maintenance simulators show quite 
diverse policies regarding, their use. Table 8 presents a sum- 
mary "of. the different types and total quantities of trainers - 
procured by'each Service and distinguishes between Naval/Marine 
Corps aviation and other Navy and Marine Corps usage. The'^ 
differences in reliance on simulation are mare evident in the 
case of standard'systems. "The bulk of Navy afloat and Air Force 
non-standard systems result from the Mk-92, Crws (Phalanx),^ 
and F-16 programs . In f the absence of these two programs, there- 
would be little difference among any of the Services. It is 
noteworthy, though, that these large programs are in areas that 
have shown the least use in the standard systems' in the past. 

TABLE 8 SUMMARY OF MAINTENANCE TRAINING SIMULATORS DELIVERED 
• AND ON-CONTRACT SINCE 1972 v BY SERVICE AND BRANCH 



f| - 


Sjtandard 


System? 


Non-Standard Systems 


Service and Branch 


Number of 


Total 


Number- of 


.Total 


Different 


Number 


Different 


Number 




Devices 


of Units 


Devices 


of Units . 


Navy/Marine Corps 


1.37 


v 354 


6- 


ii 


Aviation 










Marine Corps Ground 


27 


129 


0 


0 


Forces 










Navy Afloat ' 


4 , 


10 


25 


- . 581" ; 


Army 


24 


158 


2 


a 


Air Force 


_2 


■ 2 


11 


_§1 


Totals . 


194 


" 653 ' 


47 


687 



" One result of standardisation is that it is difficult to • 
identify specific simulator development or procurement proqrim^. 
The standard devices that provide training for a particular sys- 
tem (e.g., a qiven ipodel of aircraft) may have been procured 



through several contracts initiated at different times, ^--A- - • 
single contract may encompass varying quantities of devices for 
several equipments and include procurement of the general simu- 
lation' system as well as the unique components for different 
training applications. As a result, the discussion 8f usage of 
these devices "is limited to procurement quantities Joy Service 
#hd according to skill areas trained 'and >to the market shajres 
of four contractors, ^he non-standard systems are* developed 
/and prbcured within well-defined programs that are related to 
particular simulator systems and training applications and are 
discussed in that context. 

i - - 

1: Standard Systems 

The-£irs£ procurements of standard maintenance simulators 
occurred in the 1972-1973 period when. limited quantities were 
delivered to the Air Force, to the Navy for surface traininq, 
and to the Marines, for ground forces training. The first 
^deliveries for Naval/Marine Corps aviation* traininq .occurred 
a few years later, and since that time this training has become 
the most extensive user of standardized systems. The current 
inventory of 354 devices accounts for 70 percent of the dif- 
ferent simu^lations^and nearly 55 percent of the total units 
employed in military training. ** 

The Marine .Corps was the earliest service to contract for 
a significant number of standardized systems. A 1972 contract' 
called for deLivery of 15 units ('encompassing 11 different 
simulations) for training of ground equipment maintenance; 
this was 'followed in 1975 with a contract for 114 units of 27 
different simulations ( including reprocurement of the 11 types 
of simulations contracted "for in 1972).. The last of the Marine 
Corps inventory was del i vere Y d ,i n 1976 and none have been con- 
tracted for since that time. » 

The first known Army use of standard systems for mainte- 
nance training was in 1977, with'the delivery of two devices 
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for training of the MQhawk propeller control system. Since 
that time, the Army has procured devices foi training of,olher 
aircraft systems and several armored ^vehicles. .Note that the 
numbers shown in Table 3 include only devices that were procured 
through th& Program Manager foe Training Devices (PMTRADE) . The 
devices iisted in Table 8 may be an incomplete listing as Army 
management procedures allow training devices to be contracted 
for through weapon system program offices and individual base 
commands/' Such devices, are not registered in a /central Lnven- 
tpry. record and cannot be readily identified. t h 

Both the Air Force arid Navy afloat have made little use of 
standardized simulators. The Air Force procured one device in 
the earlV 1970s to evaluate its use in training AN/ALQ-126 radar 
maintenance personnel (as part of a, research project). A second 
device. simulating the 6883 Test Bench was delivered in 1980, 
also for >evaluat ion as part of a research program. The Navy 
procured five copies of one device for training in the tuning 
of traveling wave tubes" in 197 3. tfThe next, delivery dt this 
class of simulator ,(the Hagen Automatic toiler Control Simula- 
tori Vas in 1978 as part of a .research program investigating 
training strategies for equipments the ma intenance of which 
'requires personnel trained in different skill areas. The only 
current use of standard simulators for ma4\i-line tr-aininq con-* 
,sists of two devices for training maintenance of the Trident 
submarine air-conditioninq and air-c6mpressor systems. 

/f The standard systems. have been used for training in a 
variety of skill areas, as shown in Table 9, with training 
applications spread rather equally among the broad groupings of 
electrical and electrQnic, propulsion, and combinations of 
mechanical/hydraulic/pneumatic areas. This -.stands in contrast 
' with the non-standard systems discussed below where, excepting 
the two- large weapon-,system-or iented p-j' grams PF-16 and CIWS), 
all -but the MA-3 and DD 963 boiler prograVns have been limited 
to's imulat-lon of electronic systems. 



TABLE ,9. STANDARD 'SYSTEMS DELIVERED AND UNDER CONTRACT 
. BY SERVICE AND SKILL AREA # 



Skill Area by Miljtaey Service 



Number of 
Oi f ferent % 
Simulations 



Total 
Number 
of Devices 



Marine'-Ctfrps Ground Forces 
Electrical r 
Propu.ls ion V* A 
Hydraulic-Pneumatic " 

'Naval and. MaHne Corps Aviation 
General SkMl Training 3 

Electrical-Electronic 

Prppulsfon * 

Etectro-Hydraul it 
Weapon-Speci f-ic Training 
'V Electrical 

El ectronic 

'V^opulJiion 

El ectro-Hydraul ic 

Mechanical-Hydraulic 

Unknown * * 4 

Navy Xf 1 oa t *" <• 
^Hydraul ic-Pneumatic 
Combination of S k jL 1 1 Areas 
Electronics ^ „ 

Air Force » 

Electronic ' 
Army <j 

Aviation ->t 
Electrical- 
Electronic 

*nydrauldc 7 
Mechanical-Hydraul ic 
EJ ec trp-Mechan {jgaJ 
Ground 

Electf ica'l-f lectronic 

SropuUiod 

Hydraulic 

E 1 ec t<w>- Hydra ul ic 9' 



12 
11 
4 



10 
16 
14 

18 
23 
14 
1 

29 
12 

2 
1 
1 



4 

I* 
1 

2 

2 
4 
2 
4 



38 
75 
16 



78 
80 
76 



Includes trajnin^m aircraft and ground support equipment. 



'% relevant point to be seen in. Table 9 is. that while these 
systems have found a wider range of applications (in terms of ] 
..different simulations) for weapon- spec if ic Naval aviation train- 
ing, o m or two units (trainers) of a given simulation will > 
satisfy a training requirement. That is, 97 different simula- 
tions (training applications) are satisfied by 120 devices,, an 
average of only ohe and one-quarter units of each simulation. 
This contrasts ^with the larger numbers of" identical units re- 
quired for general aviation skill training (or for training for 
widely held equipments such as those employed by the Army and 
Marine Co rpsf'-g round forces). Non-recurring costs involved in 
bringing a simulation on-iine are high compared with the costs 
of fabricating additional units of an already designed simula- 
tion, and this relation has a large impact on the average costs 
of simulation in training f or «di fferent , types of equipments. 
• v, Table 10 shows the number of standard systems' delivered 
and under contract,' according to manufacturer. RCC appears to 
dominate the market, but -the extent of this domination is de- 
creasing. Ridgeway is a new company that appears to be aggres- 
sively marketing its system. As of mid-summer 1980, all of 
- the 107 Ridgeway devices shown .were under contract, but none 
had been delivered. ECC, by contrast, had undelivered orders 
for 100 devices. When all of .these deliveries are completed, 
the percent of devices in use that are manufactured by ECC 
will drop' from 90 to 75. j 

Standard maintenance simulators are not ma jor /P roducts of 
either Burtek or Lockheed" Aircraft, Services . Burtek produces^ 
a wide range of training devices (from aircraft evacuation 'and 
ejection seat trainers to automated study carrels) for both the 
■civilian and military markets. Lockheed Aircraft Service pro- 
vides a wide range of aircraft-related products and services 
(including aircraft modifications, full-scale models, mock-ups, 
and training services) for both military and civilian customers 
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JABLE 10. STANDARD SYSTEMS DELIVERED AND UNDER CONTRACT , 
BY MANUFACTURER SINCE 197? • 





* Manufacturer 


• 


Educational 

Computer 
Corporation 


Ridgeway 
electronics. 
Inc. 


Bur tel , 
Inc 


LUVKflCCU 

Aircraft 
Str vice 


, Marin* Corps Ground forces 
Unique Models 
Total Held. 


27 

* 129 


- 


"*■ 




Naval & Marine Corps Aviation r 










General Skill Training 
Unique HodelS 
Total Held 

* Hwpon-Specif ic Training * 
^ , Unique Models 
' Total Held 


21 

. 119 i 

74 
97 


18 
103 


1 

\l 

11 
11 


-- - 

12 
12 


Navy Afloat 

UrVique Models 
Total Held 


5 




: 3 
5 




Aroy 

Untyue Models 
Total Held 


. 22 
15? 


4 


\ 




USAr/ 

Unique Models *\ 
/ Iotah\ield 


1 . 


\ 






Total 

Unique Models 
Total Held 


• 

146 

503 


•<ttf7 

— 1_ 1 


17 ^ 
31 


12 ' 
1? 



* v. S t 

2 * No p-Standa rd Systems " v 

The non-standard simulators are relatively recent develop- 
ments* A listinq arid description of the programs that have been 
initiated to d&te are shown in Tables.ll and 12. In fact, there 
has been little experience in training with this class of simu- 
lator* Several of these programs have; yet to result .in deliv- 
eries; for several others,* deliveries have not been completed 
or deliveries have been of less than complete or full-design 
configurations. There is generally an Installation and checkout 
pe'riod and a significant period between the initial and final 2 
deliveries of a -proa ram so that, eve.n where a full configuration 
has been delivered, actual use fgx traininq would be less than 
i<5 suggested by Table 12. • ■ . " # 
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TABLE H. NON-STANDARD SIMULATOR 


PROGRAMS, 


DESCRIPTION 






Trainer Designation 


Description of Asioclated Opfcr*tlonal Equipment 


Designation of 
Operational 
tc.jpnent 


framing Equipment funtra torts]* 






Naval Aviation 












* VIM 


Visual Target Acquisition System of F-4N. 


AN/AVG-8 . 


Hcneywel 1 






ANIrainer 


"* All equipment maintained by AT rating on 
F-4. J/N and RF-4B. 


(See Table C-6) 


In-House (North Island NARF ) 






A- 6 TRAM OR* 


Detection and Ranging System of A-6C TRAM. 


AN/AAS-*)) 


Grunnan. Applied Science 






A«7 HUD lest Bench 


Heads-Up Olsplay Of A-/f 


AN/AVM.I1. 
AN/AVQ-7 


(ducati ml Co^>ute» Corporation, 
Ap'pl imaFi bn", Hofiey>el 1 , voughl, 
AACTS fngineenng 






MA« 3 Test Bench 


Aircraft I2KVA generator test bench. •« 


MA-3 


Application. ievHle 






£*>6B ICAM TJS 


Tactical Janotng System 


AN/ALQ-99 
AN/ALO-92 


Grurnan , 






Navy Afloat 

IRR 


\ 

Integrated Radio Room of Irident submarine. 


AN/B'C-1 


RCA, Ideational Computer 
Corporation 






CIUS 


Short range anli-aircr*f i gun System for 
Surface ships (Phalanx close»fn weapon 
system) 




K. 

Cubic 






Unto Heat l -6otler 


DD 963 Waste Meat Boiler 




Application 






HK 92 rCS 


fire Control System HK 9?. Mod (), 


FCS MK 9? Hod () 


Sperry 


/ 




Alr^force 












688) Test Bench 


Test bench for a portion of F— 111 avionics v 


AN/ASM -4?7 


Honeywell 




- 1 


AUACS Navigation 


Navigation yystw of 1-3A. 


AN/ASN»II8 


Honeywell, American Institute. 
for Research 






AUAOS ftjJ*r 


Radar system ot E-3A - 


AN/APY.I 


Honevwel 1 , American Institute 
for Research 






f-16 


Avionics, electrical, propulsion, hydraulic, 
v weapon control systems of F-16 




Honeywell** \ 






fRS-43t 


Ground radar system 




In-house {tester AFB) * 






^rirefinder 


Mortar and art 1 1 lery -locat *ng radar systems. 


AN/TPQ.J6. 
AN/IPQ-37, 


Hugf-es Aircraft- 






RttS 

) 


tactical cc*runication system 

V 


AN/ IRC -138. 
•AN/TCC-M (3). 
A«/JRC-M5. 
AN/TRC-lSl. 
AS-TSQ-84. 
AN/TSQ-8S 


Gould 

i 

ijJLcr_-is-^*«derlirrtf — ~ 


- 




a Where wore' than ^ne contractor his been involved in a program the name of 


the principal contjr 






«£Wco*tr*?tor to t he"*Wej porTsy s t em c o n t r a c : o r 










* 




* 

0 
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4 
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TABLE. 12. NON-STANDARD SIMULATOR PROGRAMS ,'CHARACTERI 


ST ICS 










R» interume 






















Designation 

* i 


Otvkes 




UMort 

I rj i rir J 
/0rganl2«tlorul\ 
or J 
\ ! ntrrm*JM tr / 


Purpose 
of « 
Simulator 
Progrj** 


lype 
S 1*10)4 to* ' 

/or 

Dimension*) 


Appropr w- 
Uon*> 


J*pe of 
< 0Mr«Cl 


frile of 
Contra t 
or 

PiQ9r«n v 

I.MtUdO* 


(Mte of 

InltU? 
Del Ivery «- 


Nuaber of 

DelC«ered 
Ihrouqti 
fV 1980 


Progr^a 

Cost. 
Then 
ft4r" 

fOOO! 


Pfogri^, 

1981 J 

(000) 




V 

Rivil A^ijtlon 




























VIAS 


1 




0 . 


1 


) 


P 


,,p . , 






? 


300 






At-IrifneV 




) 


0 


1 


) 


P.O 




AWS 


«»/'9 


1 


1 ,8W r 


» 2. WO 




A»6 IRAK ORS 




I 


P 


\ 


) 


t 
P % 


fFP 


V'« 




? 




K 6)0 




V; huo ieu flfftc^» 


1 


\ 


! " 


R 


3 


R 


iPf f 


6//fc 


> Iff) 


1 


1.300 


t.no 




HAO leu Bene* 


1 ' 


1 


1 * 


ft 


) 


R 


CPf/ 


9//B 


8/80 


1 


640' 


?80 7 




tA-6ft KAP US 


1 


2 


0 


1 . 


. \ " 


P 


^fVp 


1?/80 


1/8,' 


0 










































0 


I 


i 


P 


f f p 


' 9// 9 


3/81 


0 


/.090 ^ 


'.090 




ctws 


8 


788 


0 • 


I 


7 


P 


FFP 


IU/80' 


H/81 J 


0 


S.900 1 


S.900 - 




. 00963 Wute 
Heat Doittr . 


) 


r" 


t) 


I 


1 


* P 


ffp 


??/8U , 


MfH\ 


0 


><x> 


300 




H». W f(S 




.«« 


0 


1 




P 


CPff 






u 




9. boo 




Air rone 

i\ 

688J Irvl Bemh 




























1 * 


i 


1 


R 


3 


b , R 


CP», 








80-3 


1.130 




AWACS N««iqd^ton 


\ 


i 


0 






p" 


f p i r 




l,*//9 


t 




y.wo 




A AWACS Ruhr 


1 


i° 


0 


1 




p 


fpjr - 


VM 


S/8,' . 


0 








* M6 




St," 


J) 






p 


FP|« 


?/// 


i n' 


; 








|PS-4« V 


I 




0.1 


I 


\ 


C 




Ojrly/ // 


1 fte//K 


i 




120 




Amy 




























„ rirettnder h 


1 




1) 


1 


I 


H.P 


fPlf 


H/ J 1 * 


1 /^O 


:c 


i 1 } . 3«e 


^6.930 




* fifts'* . 


\ 


1 


0 


! 


3 


P * 


fPlf 




s 






4.640 




, (see^ next page for footnotes) 
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(Continued) 
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ft: Research; |; Training. 
^PS Procurement; R fiOHt. 0 Operations and Halntenance. 

C Ad}uttmenis to 1981 cost level are ba^ed on fjol) Inflation Rates (for R0IAl\or procurement) dated 4 July Ijtyt assuming (hat tti* nd-joint of tf.e progra*0 
^ expenditures falls nid»way between contract and ^nltial delivery dates \ 1 " > 

d 0ne unit contains a Capability to simulate the F-4J. f-4N. and RD-4B while t\e other units simulate only the F-4J. 

f 0oes not include the costs associated with simulation to provide a minor levei^of At rate training (estimated at $jQ0.000J 
Initially funded through RDJU and Intended for research purposes, the purpo*e.wa> changed earl/ in the program to training and furled tnrough procui ewit» 

^Includes lJO.000 for development of training Materials for a new course addressing maintenance of the MA-J test bench. * 
Also provides operator training (stabHstaent of an Em Coast training facility will require an additional unit of each type of violator 
•/letter contract. Contract value and other provisions to t>e def1nttl*ed by Hartfh ^ \ 

Prototype, t . * > , \ 

• ^Includes HOO (thousand) estimated 1 or Maximum Funded itcrts. \ 
1o be itemined x 

"Cost to the government. Includes contractor fees but riot unreimbursed overruns \ ~ I 

"(he one device consists of a central processor and Instructor station, control ling 10 studeM stations 4 

°flot Including program support effort to be provided by American Institute for Research. .< « 

P J0 devices delivered to USAF the regaining 26 devices are for. delivery to NAI0 countries ,fhe cited *.ost included all i6 unit* " 

^Iwo devices are cockpit «oU*ups"whl1e the remaining |6>are flat panels. \ 

Oellvered a* interim configuration, limited *o demonstration of system operations, that will b<\ field retrofitted for malfunction lapaHhties - 
* s 0«e unlVwas delivered in late 1<>78. A second unit has recently entered fabrication \ 
first unit only. \ 

u flve complexes of si* student station. Threv complexes provide cnly operator training, two corepleos are configured to trovide both operator and 
maintenance training ♦ v 

v Ihe delivered trainers do nut »>«*ulatc the cuf»eht umf tgui ation gf opetattfaal njuuwrwnt and will bet eti of * tted in the MelJ ,V 

in di .pute \hv initial c*'»tiait *alue was (thousands), but the contractu* hj> t*»<*'«tl» Hied a claim *or an addittOMt st» million* 



* These 1*7 programs do not seem to provide a representative 
sample of the potential range of application for the non- 
standard ^simulators. All but the three research programs . 
(A-7 HUDf MA- 3 , and 6883 Test Benches) are concerned only 
with organisational maintenance, and these three were devel- 
.oped as* research vehicles. Initially, nQne were envisioned, to 
provide main-line training, although , current* planning is for 
the MA-3 to provide training. Only four programs (MA-3 Test 
Bench, CIWS, DD 963 boiler) s imulate other than electronic 

equipments.. The F-16 and CIWS are large simulation programs 
to provide training, in several of the skulls (including elec*^. 
tronics) required for maintenance of a weapon system. This 
seems hardly representative of overall maintenance training 
requirements and is quite different from the pattern observed , 
for the standard systems. 

a. Concurrent Development . The .sample is sufficient, 
though, to illustrate some of the characteristics associated 
with non-standard simulators, three of which are .discussed here. 
The timing of deliveries of training devices is critical for 

the introduction of new or modified operational equipments. 

> 

Training equipment, of whatever type, must be in place before 
•training can commence, and personnel must be trained before 
the operational equipment can become an effective" part of the 
'force. Training simulators -equire their own development 
periods and this must occur concurrently with development of 
the operational equipment. However, the operational equipment 
is subject to frequent modification during development and for 
a considerable period after its initial fieldinq. Even minor 
modifications can have a larqe impact on the costs of simulator 

• or 

.development. At least five of the 17 simulator programs in- 
volved concurrent development; the A-6 TRAM DRS, the A-7 MUD, 
the Trident Radio Room, the F-16 trainers, and Firefinder. In 
each case, the simulator programs incurred significant engineer- 
ing changes that increased their costs. The A-6 program required 



^extensive software changes, that .amounted to 30 percent of the 
Winal program .cost* The addition of FLIR to the A-7 HUD traiher 
accounted for approximately 40 percent of the u total program, 
cost. In the F-16 program, it seems impossible to attribute a 
dollar east to changes, but they are generally acknowledged to 
bo a major portion of a cost overrun that amounted to .three 
times the initial program estimate. Not only were Changes to 
the aircraft frequent, but documentation of the changes that 
were nijeessary for simulator design ran as much as 12 months 

behind implementation* of the chang.es themselves., Changes in 

a, 

the Trident Radio Room and Firefinder program were^not'as dra-* 
matic but still had a sigr^fcficant impact on development costs. 

A related problem is that modifications and configura- 
tion changes are common for aircraft that nave been fielded for 
a considerable period. ^Changes to operational -systems may re- 
sult^ in simulator modifications whose costs approach the cost 
of development of the original device. This is close to .the 
situation of the A-7 HUD simulator; a day version of the opera- 
tional equipment had been in use for several years, and the FLIR 
version entered development during development of the trainer. 
Contract costs attributable to modifying the trainer to simulate 
th£ FklR ampunted to 85 percent of the original contracts for^ ' 
the day version trainer.. Modifications to operational equipment 
have resulted in the obsolescence (and discarding rather than * 
modification) of a number of standard simulators. x 

b. Quantities Fabricated .' For nine programs in this 
sample, for which cost information was available, development 
cost averaged over three times tfie recurring cost of simulator 
fabrication and initial support. This provides a large' poten- 
tial for reducing average costs by simulation of equipment^for 

which there is an e/tensive training requirement, suc'i as equip- 
* • s . 

ineht, used* for general skill training and .equipment that is used 
on widely held' weapon systems. Orjly three of the t i*7 programs in 
the sample simulate this type of equipment. The MA-3 Test Bench 



is a universal tost stand used throughout the Navy fot onsfhQre 

testing of all models of generators and constant speed drives 

that Supply aircraft primary electric^ power,; although the MA-3 

Simulator is a- research device, with only one unit built/ it 

iias 4 a potential for providing training at.ali Naval and Marine 

Corps air stations. /The CIWS and Mk 9-2 Fise Control system 

are to be installed on a large number of surface ships, genet^at- 

ing an extensive training requirement; current planning cails 

for fabrication of 36 sets of eight simulations for' CIWS train- 

ing and 24 sets of 12 simulations for Mk-92 FCS training. In 
* " » » 

contrast, a few units appear to satisfy the traininq requirements 

for t^e. bulk of ^the other simulators in the sample. As examples, 
thq present F-16 contract provides for delivery* of equipments^ 
to only threq, air basefcs; training for specific types of Naval/ 
Marine v Corps aircraft is provided at only orte to three ai.r sta- 
tions so that buys of weapon-specific simulators (such as the ; 
A-6 TRAM DRS) will be limited to'a small number. 

c. Substitution Relations . Simulators are, generally 
viewed as \subsfc i tijtes for actual equipment trainers. Whether, 
this is a correct wqy to view simulators should be argued 
separately; simulators and actual equipment can each bemused 
Cor traininq in ways that #re not possible by the other. The 
auestion of substitutabi 1 ity is not a simple v> one and the extent 
of . substitution depends on the nature of the simulation, the 
equipment being simulated, and the extent of traininq provided. 
Within thfese 14 programs are examples of four different rela- 
tions between simulators and actual equipment trainers. 

- » t . » ' . > 

The MA-3 Tost Bench and 6883 Test Bench programs il- 
lustrate cases approaching pure substitution. Each program 
provides traininq in both the operation of a test bench (i.e., 
maintenance of operational equipment) and- riaintenanco of tho, 
test bepth itself. Fach simulator was desiqned to replace 
some (but not: jieces^san ly al 1 ) operational equipment bl^at had 
been used for traininq. 

* ' % * 
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In the case of the AWACS system, the navigation system 
simulator has been used to implement a substitution of school 
training fo" on-the-job training (OJT) ♦ 'Prior to introduction 
of the simulator, formal (school) training was limited to the 
classroom, and hands-on systems training was provided only as 
OJT at an operational base. Introduction of % the simulator 
permitted the hands-on training to commence at the^training 
school £nd. should result in a shorter period of OJT before 
personnel are qualif iedv for independent work. - 

Trident radio room training employs bQth the Simula- 
tors 5 and a modified complete operational rajdio room (actual 
^equipment trainer). That is, they complement one another, 
with each contributing to different elements of the curriculum. 
They are alsb substitutes. An early assessment of. Trident 
training requirements developed two alternative equipment 
configurations ftfr radio room training. One was the current 
combination of simulation and AET. The second was the use 9f 
three, AETs only. The choice of the combination of actual equip- 
ment;and simulation was Based, at least in part, on cost con- 
siderat.ipns. * 

in the case -of several other programs, both actual 
equipment and simulators are used, but for somewhat different 
reasons. The-, A- 6 -TRAM DRS contains both electronic and mech- 
anical componerlts. 1*he simulator is limited to training on 
the electronic p^kion of the system, and the actual equipment 
is required for the mechanical training. None of the organiza- 
tional echelon aviation simulators can wholly substitute for 
operational equipment. Typically, both organi?,ational and* - 
intermediate maintenance training is provided at the same 
location and, frequently, in the same training course. The 
'intermediate level training will require actual equipment, but 
normally in the form of individual components rather than aru 
integrated system. 
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The cost effectiveness <ot simulation/ for training 

maintenance skills depends upon its impact on/ total training 

costs. Variations in the substitutability of simulators for 

* / 
aqtual equipment trainers imply that simple /comparisons of the 

' / 

relative costs of the two types of training/ equipment cannot 
be taken as reliable guides to the rela&iv^ costs of training ♦ 
Assessments of the cost advantage of usina simulation must be 

based on comparisons of the total costs ok satisfying particular 

9 / • 
training requirements with and without such simulators. 




D.* STRUCTl/ftE /FOR COLLECTING COST DATA f 

The set of cost elements shown ih Table 13 is an initial 
formulation c/f a functional cost structure for collecting data 
to develop Cjt>st-estijnat ing relationships and other tools for . 
assessing the costs of maintenance training simulators. It is 
a mixture 'e^ elements that are generally associated with other 
types of military equipment as well as those that seem particu- 
larly relevant to ^tfrocfcssor-dr iverf simulators; it relies heavily 
on discussions with people who have had, experience with simula- 



tor procurements. ■ \ 

This/ cost element set is incomplete .in two ways. First, 
it is at la level of aggregation that may prove insufficient for 
identifying the basic cost drivers. Second, even at this rela- 
tively high level of aggregation, w& are uncertain that it is 
f ully'specif ied. with our current knowledge regarding the 
determinants of cost, it does pot appear feasible to carry the 
specif ica^t ion further. 

This\ structure does, though, treat the two important cost 



characteristics of maintenance training simulators evident in 



the data currently available/ — ttfe separation o'f recurring from 
non-recurring costs to identify program development costs and 
the separation of software (/and courseware) from other develop- 
ment costs tq identify the (apparently) dominant requirement. 



ible /- 
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TABLE 13. COST OATA STRUCTURE 



Front End Analysis 

* Task Analysis 
Performance Specification 
Engineering Specification 



Design, and Development 
Hardware 
Software 
Courseware 
Technical Data 



Test and Evaluation 
Acceptance 

Training Effectiveness 



Fa bri cation 
Hardware 

Installation and Check-out 
Special Tools/Test Equipment 



Logistics Support 

Interim Maintenance Support 
Other 

Facilities (Construction/Modification) 
Initial Training 
Program Management 
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Further, it is structured as a matrix* Several contractors 
may be involved in a sinqle program; one contractor may be 
engaged in several (sequential) contracts on a single program; 
programs, are generally sectioned into distinct and identifiable 
phases; and, changes in program scope and statements of work 
are frequent* In each case, the types of work and relations 
among costs may differ in systematic fashion, and these dif- 
ferences should be preserved in whatever data are collected* 

A maior problem with formulating a structure for collect- 
ing cost data at this time is the current paucity of data* 
There is no general requirement for systematic and periodic 
reporting of all elements of the costs incurred by contractors 
of training equipment* With a sinqle exception, standardized 
work breakdown structures (WBS) for training equipment have 
not been developed and employed; thus even if contractor costs 
were to be reported, there would" likely be i ncompatabi 1 i t ies. 
among the data from different programs* ' / 

A periodic cost reporting system addressed to simulatdrs 
should be based on a sinqle basic WBS that would be applicable 
to a variety of simulator types and other training equipments 
and serve both proqram-manaqement and cost-assessment functions* 
The Army is currently developing a general WBS for all training 
equipment* Tt has yet to be imposed on a procurement program, 
and it appears co be directed only to cost assessment* The Air 
Force has developed a WBS that has been used f,or botjx^man 
and cost assessment, but 1 1 supplication is limited to flight 
simulators* There ayuetrT~Tb be s igni f icant t d 1 f f erences between 
those two^-fert'TTT^TuTes , and neither seems to satisfy the criterion 
oT^qeneral applicability (o*q*, neither appears to provide for 
a definitive separation of recurring and non-recurring costs)* 

In general, the program costs collected durinq this proiect 
(contained in Appendix B) are assessed in t'he format of Table 13* 
A next step in assessing the format (i.e., the adequacy of data 
it displays) would be to obtain measures of simulator physical , 
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and performance characteristics for the current programs, to 
test for relationships between these characteristics and levels 
of costs and, since this structure addresses only assessment df 
costs, to reconcile it with the data requirements of-program 
management. ^ 

E. COSTS OF MAINTENANCE TRAINING SIMULATORS 



1 • Standard Systems '~~ — ■— I— ^^^^^^ 

For practical purposes, the data now available on stand- 
ardized systems are insufficient for analysis of cost and es- 
pecially for relating costs to physical and performance charac- 
teristics of the trainers. Almost all procurements are. unde 
FFP contracts where formal documentation is typically limit/d 
to perCormance specifications issued with the request for pfco- 
pcsal (RFP), contractors' technical proposals, and the con/ract 
itself. The physical and performance characteristics may 
change, as the result either of contract negotiations or Li 
subsequent contract modifications, with the correspond! nd docu- 
mentation not being revised. 

^Vost documentation is norma^y limited to the line/item 
structure of the contract; for- the standard systems thik is 
sketchy at best and can be misleading. A major problenf is that 
contract line-item structures ar(/ in terms of the prod icts (or 
deliverables) that result from the contract (e.g., trainers, 
data, contractor field services, conferences). While this 
structure does provide useful information for cost conltrol and 
management, it provides none of the attributes of a functional 
WBS necessary for evaluation. The contract line item "\rainers" 
typically encompasses over 70 percent of a total contract^ value . 
Within this 70 percent are contained (or hidden) those cosi dis- 
tinctions that allow simulator and procurement program charac- 
teristics to be related to program cost (e.g., between recurring 
and non-recurring costs, between development and fabrication, 
between hardware and software). 
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Nine contracts were reviewed, and the information they con- 
tain is shown in Table 14* (In the discussion below, these con- 
tracts are referred to according to the column number of the 
table.) This set of contracts includes four models of standard- 
ized systems built by three contractors and appears to present a 
representative sample of available data* 

Individual contracts show a wide range in the number of 
different types of trainers or simulation models developed (1 
to 27) , the number of trainers procured (2 to 114), and their 

average "costs (from under^TPrftfr^M^e^^ An impor- 

tant feature for assessing costs is that most contracts involve 
development and procurement of trainers for a number of training 
applications and several copies of each type of trainer. 

The line item listing shown in Table 14 is close to the full 
cost detail given, in the contracts. The only items contained 
in the "other" cateogry are conferences, training, and reliabil- 
ity and maintainability programs and demc istrations. A separate 
line item is normally shown for each type of trainer delivered 
on the contract, but that sinqle line will contain th/ cosP^ 
both the first or prototype unit (with the development costs \ 
it entails) and all follow-on units. 

Separation of costs between the prototype and follow-on 
units is contained only in contracts 1, 6, and 9. In contract 3 *** 
1, the same unit cost is charged to all 101 follow-on units; in 
contract 9, follow-on units (not included in the^Table 14 values) 
are specified as a contract option at a cost dft ferent. from the 
prototypes, but four of the six trainer types are attributed 
with the same follow-on unit cost. In three otTibr contracts, 
the same unit cost is charged to several different types of 
trainers (both prototypes and follow-ons). In contract 1, the 
average ratio of prototype to follow-on unit costs is approxi- 
mately 16 to 1, while in contract 9 the ratio averages 3 to 1. 
Since ECC is the contractor in each case (although different 
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TABLE 14. STANDARD SIMULATOR CONTRACT INFORMATION ANO COSTS 

(NINE CONTRACTS) 



SeMce 



fiscal fear of Contract 

Simulator Model 

Type of Contract 

Contract Value (OOO) 

Number of Tramers Procured 

Nutter of Simulation Models 
Developed 

Average Nus&er of Trainers o* 
Each Type , 

Average Contract Cost Per Trainer 
"¥'* <r- i _ 



Average Cost of Trainers (OOO) 

Range of Unit Cost of Trainers (000) 

Contract Costs by line Item (000) 

Trainers (including installa- 
tion}' 

Technical Oata 

Interim Support 

factory Repair of Spares 
and Paris 

Contract Field Service 

Spares and Spare Parts 

Support and Test equipment 
and Tools 

Logistic Support Analysis 
Other 

Contract Total 



Navy 
1975 
CCII 
FFP 

n.i32 
11 



• 3 



S9 9 



S9Q1 
142 
19 



Navy 
1978 
€C3 
FFP 
SI, 301 
91 

14 

t s 
SU 3 



3 



Navy 
19/9 
Ridgeway 
FFP 
SI, 131 
103 

18 

5 7 

sn.o 



S8 9-16.9 

Sl.:68 

15 

S 



10 
1.132 



1 ,482 



$6 S-9 0 

S805 
. 27 

;$ 

4$ 
40 
SO 



24 
1.131 



Navy 
1978 

Burtek 
FFP 

S2S9 
2 



1 0 
S129 5 



SS6 & 61 

sn; 

*3 

l? 



3 



8 

:s9 



Amy 
197.8 
£C3 

i;. 

3 4 
$3? 5 



S23.8 



$410 
18 
bO 



:o 



Arwy 
1977 
EC 3 
FFP 
SI, 770 
72 



10 3 
S24 6 



7 



S18.3 



SI. 363 
244 
70 

10 
12 



11 
552 



$2 
1.769 



Amy 
1978 
F.C3 
FFP 

SI, 556 

28 



25 

S5S 6 



S34 o 



SK 06$ 

304 

75 
MO 

10 
29 
13 

1.663 



Amy 
1979 
Ridgeway 
FFP 
S236 
4 

1 

4 0 

SS9.0 



S106 
18 



70 
30 



8 

236 



.9 U 



•— Arry 
1979 
F.C3 

rpir' 

S2.651 
13 ' 

6 

2.0 
S203 9 



7oZ & 



SI. $48 

293 



*3 



43?^ 



Several contracts have undergone edifications Where information was available, the values m »Ms table reflect the 
.-nodifications. 

^Contract included ore hands-on trainer with a cost of S567 thousand and 12 simulation trainers Cost of me hands-o* 
trainer is included in ;he costs below, euept as noted 

c P r'oc ur ere nt included 16 III*, ^on^oles. 13 different simulations lb6 total devices) developed on this contact, 11 different 
simu'aticns [4$ total devices) developed on an earlier contract, and three devices, addressing basic steals, that were 
developed by the contractor for the vivil nariet. 

deluding cost o'f the hands-on trainer 

e The ranqe of prototype (first jPit: costs was S2T 9 to Sl2 2 thousand All follow-on units were priced at $1 D thousand, 
regardless of whethe. the Jevt.e wav Jeveicped in this contract or the earlier <. -ntract. consoles were priced at S16 c 
thousand each. 



Contract lines 1 tew listings normally S nc* ea*.h type of trainer and its tosts as a single i 
of all trainers of one tyi-e will genera'ly contained in that single entry 



ic: ite* however, the'costsj 



1 llfi l !2 r iJJI?J!l 0 !}* B # ' 0r . c I a, "» "on contract nodincations and S241 thousand for extensions to the software 

_ sys^c^escrjbed as for test set(s). procedure and performance nonitoring. , \ . so^ware 
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standard systems are involved), we wbuld expect that the dis- 
tinction between recurring and non-recurr inq functions would 
be thfi_ same f oc„cach_ con±.ract^ II this is -true, the wide vari- 
ations in the recurr inq/non-recurr ing ratios are difficult to 
accept. In only two contracts, (1 and 6) are the costs of main 
frames and panels shown separately, and the ratio of main, frames 
to panels varies. No contract provides for a separation of 
functions associated with development (e.g., hardware, course- 
ware) except for technical data, and in this case a single line 
entry applies to all trainers included. in the contract. 

There is a considerable difference in the structure of 
costs among the contracts, and it appears that the meanings of 
^i»bM^t- — i-t-p<u^_j jnr >m flAmcg ha ve not bee n consiste ntly applied. 
For example, contracts 2 and 3 involve deliveries (by different 
contractors) of devices that satisfy the same training applica- 
tion and have display panels constructed to a sinqle specifica- 
tion. On the basis of the ranges of unit costs that are given, 
it would appear that KCC costs are hiqher. However, on the 
basis o{ average contract cost per delivery (total contract 
Value divided by the number of trainers procured) the difference 
is considerably narrower, and it would appear that a number of 
contract functions that are costed separately in the JUdgeway 
contract ate included under the cost of the trainers in the 
KCC contract. 

This discouraging assessment of available data has been 
reinforced by discussions with procurement office personnel at 
th\^Maval Training Fquipment Center (NTEC) . Several have 
expressed opinions on tv»o points that impact on the validity 
of contract item costs. One is that contract neaotiators focus 
on "bottom-line" (total) costs and that, within this constraint, 
contractor tepresentatives will trade-off the amounts charged 
to individual lino-items until the relationships among them 
look "reasonable". The second point is that contractors l, ;ve 
an incentive to inflate the cost of simpler devices and to 
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deflate the cost of the more .complex devices* tn this way, as 
contractors are successful in delivering the simpler devices 
early in a contract, they can speed up their receipt of progress 
payments relative to actual expenses* One result of these prac- 
tices is that the relationships among different elements of £ 
contract costs will be distorted, and providing more detailed 
cost statements will do little or nothing to provide accuiate 
relationships between physical and performance characteristics 
and costs* 

- *" 
2* ' Non-Standard Systems 

The program costs discussed in this section are based on 
nine programs for which information was either received in or 
trans lqtedJLojthe_ forma t shown in Table 13* These programs 
are discussed briefly and their individual co^tlPdTspTayeS in ~~ 
Appendix B. 

The cost inrormation com£is^|rom two sources* One consists 
of contracts and contractors 1 proposals; there is no way to 
determine' if the level and structure of costs contained in 
these early estimates did occur* The second source is program 
office estimates of incurred costs based on the records and the 
expertise of program office personnel* 

In either case, there is no way to compare these estimates 
against true costs* Simulator programs fall below the cost 
threshold of major procurements for which contractors are re- 
quired to submit periodic reports in a prescribed WBS* Contrac- 
tors employ different terminologies; the structure of their 
accounting systems differ, and there is an ever-present possi- 
bility of. misinterpretation in translating the available infor- 
mal ion into the categories and format shown in Table 13. Con- 
sidering the wide range of possible differences among simula- 
tors and simulator procurement programs, we question whether a 
sample of nine programs is satisfactory. However, it does pro- 
vide ii sights into two important cost characteristics that # are 
discussed below* 
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' Table 15 shows the percent distribution of program total 
cost* according to cost element, in terms of the lowest and 
highest observed percentages in this table and the average of 
the percentages. Note that the percentages have been normal- 
ized in the following two ways: 

1. Recurring production costs have been adjusted to the 
level of costs that would have been incurred if only 
one unit had been produced by dividing recurring fab- 
rication costs by the quantity fabricated. This ad-, 
justment provides a consistent base for the relation 
between recurring and non-recurring costs. • 

2. The MA-3 and 6883 Test Bench research programs incurred 
significant costs for evaluation that were not included 
in calculating the. test and evaluation percentages in 
order that all simulator programs might be treated as 

. JLJb^ugjl_ they were intended for main-line training. 

Two' distinct patterns emerge from this small sampled The 
first is the consistently high proportion of total costs that 
are devoted to the non-recurring functions (primarily design 
and development) when small production quantities are involved. 
Further, the average recurring production cost (18'percent) is 
probably overstated as only the AT Trainer and 6883 Te.st Bench 
programs identified the non-recurring* portion of fabrication 
cost that, in these cases, averaged 40 percent of the first 
unit recurrinq fabrication cost. 

FigVire 4 is a plot of the non-recurt inc, percentages when 
program costs are adjusted only to exclude evaluation costs 
oL the MA-3 and 6883. The outlying high point is the AWACS 
Navigation/Guidance system program, and there is no explanation 
why the percentage is this high. The outlying low point is the 
Visual Tarqet Acquisition System (VTAS) program. A review of 

*Tooling, planning, and the other requirements normally charged 
to production accounts that do not increase with quantity. 
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TABLE '15. PERCENT DISTRIBUTIONS OF COST BY CATEGORY TOR 
EIGHT NON-STANDARD SIMULATOR PROGRAMS (NORMALIZED) 3 



Cost Category 


Lowest 
Observed in 
Any Program x 


Highest 
Observed in 
Any Proqram 


Average of 
Observed 
Percentages 


Non-recurring Costs 






• 


Front End Analysis 


0 


18 


8 


Oesign and Development 


34 * 


81 


54 


Hardware 


2 


jj 


1 


Software/Courseware^ 


12 


53 c 


31 


Technical Data 


0 


21 


6 


Hardware Fabrication (Non^recurring) d 


4 


6 


5 


Test and Evaluation 6 


1 


3 


1 


Program Management 


3 


-24 


11 


Total Non-recurring 


61 


92 _ 


78 


Recurring Costs 

production — " 








Hardware Fabrication 


4 


36 


18 ' 
15 


Other 


0 


9 


4 


Logistic Support 


0 


13 f 


5 ' 


Initial Training 


0 


4 


L 


Total Recurring 


8 


39 


25 



Percentage Distributions 



^Recurring production costs were adjusted to reflect a production quantity of one, 
test and evaluation costs of the research programs were not included. 

Data on several programs did not separate software and courseware development 
costs. In this table and remainder of this section, these cost elements are 
combined and referred to as "software/courseware." 

c 

The high percentage case is a program that incurred software problems becaube of 
concurrency. The next highest program incurred 42 percent of total costs for 
software/courseware. 

^Based on two programs. 

e Based on si* programs. 

^The two highest percentages arose from (1) development of a complete depot 
maintenance facility and (2) over three years of contractor maintenance 
during an extensive evaluation program. The next highest percentage is 7. 
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FIGURE 4. Non-recurring cost as a percent of 
program total cost according to 
quantity fabricated 



the VTAS .simulation ami teaturos of the procurement proqram 
suqqests that it* is different in some fashion from other cue- ' 
rent (and ptobably future) non-standard simulator proqj<tfns. 
Other than the research and i n-house proqrams, VTAS was initi- 
ated 2 years earlier than any other non-standard simulator. 
Also, the avionics system simulated was quite simple by the 
then-current standards and the traininq requirement was simi- 
larly simple. These cons idernt ions sugqest possible differences 
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in the. mix of resources employed and the contractor* s organiza- 
tion and management of the^program. In either c ie, the differ 
ences would impact on the percent distr ibut ions. 

The A-6 TRAM DRS program incurred a large cost for soft^ 
are changes resulting from changes in the operational hard-* 
wars?.' When the change costs are disregarded, the percentage 
is quffce consistent with theother programs. The split be- 
tween recurring and non-recurring costs was available /or the 
F-16 s imulator program.- The percentages associated, with this 
program (the shaded area) are consistent with the pattern of , 
the Qther six. It would appear that production quantities 
of five and over are 'required before recurring costs/will 
equal non-recurring,, costs. / 

The second feature to emerge is the high cost/of develop- 
ing software and courseware. Within this* sample, the combined 
cost of software and courseware averages over 30 percent of 
total program costs (as adjusted to reflect production of one 
unit) and over 40 percent of total design/development costs 
(with a range from 17 to 72 percent). When software/courseware 
costs are plotted against total costs (not ad") us ted for the 
^pXQdiictJ^njjujw^t ity) no dfstinctive relationships are evident 

(Pig. 5). ~ ~ ' ~~ ' - — 

We have no explanation for the absence of. an orderly 
pattern or for the wide range of observed percentages. This 
small sample contains programs with diverse characteristics 
and, on a case-by-case bajsis, a number of reasons appear 
plausible. A likely reason for at least part of, the ranae 
of values' is differences in accounting practices among con- 
tractors. It is also possible that x>ur Inability to separate 
software and courseware serves to obscure underlying relation- 
ships that may be present. Further data ani analyses will be 
required to provide any understanding of the determinants of 
cost.- Consider ing " the magnitude of the costs In this sample, 
such data and analyses hre warranted. „ 
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. F I G U R*E 5. * Sof tware/cour seware *cost as a percent of 
— ^ prograaL iotal cost* according to quantity* 
fabricated ' . 



It is? unfortunate jt!iat available data do not permit* the 
separation of software and courseware development costs tor , . 
foe the standard systems, the separation of software and course 
ware from, harvdwa're development and production costs)* It is 
quite evident, at least for the non-standard systems , that 
software and courseware are significant cost items* A relevant 



7$tf#$Vij6a, then* is how these ^costs can be reduced for, future 
simulation j system's* One promising avenue appears to be the 
<Seyeioj>m6nt of a* single software system with the following 
afc^i^tj&S::* (ih it, would be non-proprietary jte.g. # owned by m 
tJie*gOY^ri|ment # (2) it would permit cpurs^warje to be written 

Xti a^higtHlevel language that could be composed and/or modified 
by?l* *9» >{-&v-hOUSG subject mait c er experts, (3) it woaid be suf- 
ficiently generaLth^t its usfe coul<? fee imposfed a contract 

/conditioner design pa^am^ter/ .The development ,o£ the Ada 

'language (may Jcfe a, large step in this -direction.* f 
; . Ipfcis concept 1$ not without problems of implementation- 

JK^is^ly reasonable to expect contractors t o resist u se of 

7®&ch '.fion-propyctotary systfem. By definition,, the standard 
't ^sterns Employ proprietary standardized s^f twa£e* c Some of the 

l?^IScu>rs of :non-sv,and&rd syptffms Appear to Have pat efforts 
intp dcvfcVoping-'*tf$ieir own &oftyar6 systems.. In both cases, 

tractors have expanded -assets in the development of these 
$oftyag^ ^iHckaqo«s,, the values' of which would be greatly dinun- 

• m ? v Figures 4 a»d t 5* Cocas 6n one problem, in identifying train- 
Tng\^ogriims as candidates for simulation. Judging by the cur- 
Vent non-standard, su^alator pcograrrs, most maintenance simulator 
appUcat iofU» A *rifin, in vsystem-speclf ic training and particularly 
in aviattW£traiofcng.>v Vhiu type of traininq is generally pro- 
vjUlocJ at, a small number of sites and requires a limited timber 
t 0l> tF*A!un$ devices, implying a limited potential U>v uuantity 
production of 4 pfirv&^ular model or training simulator and thus 

limited opportunity fcr reducing simulator costs throuqb their 
vfcdVftptedd adoption. From a cost Standpoint, the more promising 

, employment* l\e ,tn the triiaittij of general skill', and system- ; 
sjpi*CiCit; twnlpg J«r wwjely. held eqaijpments where a relatively 
large numbrr df simulators can he 4sed v 



F. SUMMARY 
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For assessing cojts maintenance training simulators, it 
is useful to distinguish between what are defined as "standard, 
non-standard ,, M and "CAI-l iJcte" sytems. Differences among these 
three classes lie in the following areas: 

• Physical characteristics, 

• Complexity and cost, 

• Extent $*f, } use within the\ Services, and 

• Contracting practices employed for their procurement. 

A standard system conjsists of a standardized physical configura- 
tion that can ,be adapted to many trailing applications through 
courseware an^'pTcTd^cfial representations that are Tailoi-ed^to 



Non-standard systems 



the particular equipment being simulated, 

a-se typical ly-unique^ in totzdr, fcrr^cich specific training 
application. A CAI-1 ike 'system typically uses a 2 -dimensional 
display (e.g., CRT, /random access sl.ide projector) to present 

^ments/through 

computer , 

Available cost/data are not adequate for developing tfre^ 
cost relationships 'necessary for comparative assessments of 
alternative maintenance training simulators. F/k:tors contribut 
ing to this condition are as follows. 



lesson materials, arid can simulate dif ferent^e^L^me 
courseware introduced into its computer. 



• Simulator pjrcrams fajl below the ca£t threshold for 
periodic reporting of incurred dosts in a standard WBS. 

• TJfe datfa that are available may .contain systematic 
biases so t^iat their reliabilyty 1 may be questioned. 

• For the standard systems f irA-t i>ied-pr ix:e ( FTP) Contract 



have been prevalent; the orfly generally available cost 
information is limited »/5 precontract documentation and 
the cbntra,cts themselves . 
• Within Jthqf small number of non-standard systems that 
have been built, tWere has been al'wicie range of program 



arrangements and purposes, device complexity and char- 
acteristics, and training capabilities* It is doubtful 
whether this small number would provide a satisfactory 
base for developing a cost-analysis capability* 
• For practical purposes, cost data on CAI-like maintenance 
— simulator system costs are not available* Only one 
experimental system has been built, and contracts for 
prototype development of two other systems have only 
recently oeen let* 
There is an obvious advantaqe in the concurrent develop- 
ment of operational and training equipments so that trainers 
are available at the time operational equipment is first 
f ielded* - -However r - th i s przlet i-ce en t a i 1 s r i sk~_ si ace— the, opera.- 
tional equipment is subject to continual change and even minor 
changes may result in high cost modifications to training 
simulators (especially to the simulation software and course- 
ware ) ♦ 

For the non-standard simulators, non-recurring costs 
account for the maior portion of total costs when production 
quantities are small (e*g*, five or less)* However, most 
potential applications appear to be in weapon-system-specific 
training (especially in aviation) where a limited number of 
devices would be required* 

For the non-standard simulators, sof tware/courseware 
(i*e*, program design and programming) appears to be the* 
single largest element of cost. Where cost overruns have 
occurred, they appear to have been primarily due .t6 software 
development problems* There should be a significant cost 
aclvanta,e gained by development of a widely applicable and 
non-proprietary software system. 

/ 

/ 



IV, COST-EFFECTIVENESS OF MAINTENANCE SIMULATORS 

The crucial question is whether maintenance simulator^ are 
cost-effective for training military technicians. Since ost, 
effectiveness, and cost-effectiveness are not, in themselves, 
absolute quantities, thi^ question must be answered in relative 
terms, i.e., compared to what else is a particular maintenance* 
simulator cost-effective? All of the studies with relevant data 
compared the cost and effectiveness of maintenance simulators to 
t hat of act ual^qu-iproen^tr<vi-ne-rs-. 

With respect to effectiveness, the data from 12 studies 
show that student achievement at school is^about the. same Jor 
those trained with simulators as for those trained with actual 
equipment trainers; there was one case where students trained 
with simulators had poorer achievement scores^. We would prefer 
to estimate' the effectiveness of maintenance simulators and of 
actual equipment' trainers by comparing the performance of 'tech- 
nicians (trained with one or the other) on the' job rather than 
just at school, job performance could bo n^asured by data such 
as^ the time needed to identify malfunctions and to repair or 
teplace faulty components, the number (or percent) of repairs 
where qood parts were removed unnecessarily or bad parts not 
identified and so on. 

No evaluation of a maintenance simulator reported objective 
job performance data. In oae stuHy, supervisors 1 t -itinqs (i.e., 
subjective data) showed about the same level of job performance 
for technicians trained with the 6883 Test Station 3-d imensiona 1 
maintenance simulator or the actual equipment (Cicchinelli et ai . 
1980). Based on the data on student achievement at school and 
the one case of supervisors 1 ratings of on the -job performanre, 



we conclude that maintenance simulators and actual equipmen^^ 
trainers e>e equally effective for training maintenance tech- 
nicians. 

This finding is based on a wide spectrum of simulators, 
i.e.* maintenance simulators of radars, vehicles, electromechan- 
ical equipment, 2-D and 3-D-designs, and simulators that are 
used for training organizational and intermediate maintenance. 
It would be tempting to infer that one type of simulator or a 
particular way of using them, among these classes, is more ef- 
fective thar; another." No such breakdown, appears possible with 
the limited data available. We cannot answer such interesting 
questions as how effectiveness might vary with cost or how 
cost might vary with effectiveness, because no such trade-offs 
h ave been ^^ertake n* _ We toyj^ of the 

costs of maintenance simulators and of actual equipment trainers 
that have been 'shown to have equal effectiveness for training 
at-. school . So, we^arcT Ye f t~ with TR~ge7^"a 1 concTusTdn , a s 
stated above, that maintenance simulators and 'actual equipment 
trainers art equally effective for training feeenn ic ians . 

^ Our evaluation of cost . uses the cost data presented in 
Chapter III? those describe acquisition but not life-cycle 
costs. The costs of acquirinq actual equipment or simulators 
do not include the costs of their use. for training purposes, 
e.q., the operating costs of training such as instructors, 
strdent pay and support, maintenance of training equipment, 
and management of the school. A cost-effectiveness evaluation 
based on acquisition costs alone must ..be regarded as incomplete 
compared to one that includes all life-cycle costs. A sinqle 
exception, in the case of the life-cycle cost comparison of 
the 6883 Test Stand 3-D simulator and actual equipment trainer, 
reported by Cicchmelli et al. ( 1980), will be considered 
separate ly . 

, Table 16 shows the acquisition costs of comparable simu- 
lators, actual equipment trainers, and operational equipment 
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TABLE 16. 



ACQUISITION COSTS OF COMPARABLE MAINTENANCE SIMULATORS, ACTUAL EQUIPMENT 
TRAINERS AND OPERATIONAL EQUIPMENT NOT CONFIGURED FOR TRAINING 
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(see next page for footnotes) 
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TABLE 16. (Continued) 



Note; All costs are om " theo-year" bast* Costs obtained frc* the Consolidated Kanagwent Data List '<P "em laf^' ontrj. t -osi 

but do not reference the applicable years 
*Where we than one so*.,ce was avall»ble the figures shown-* re the highest values found 

b Recurrlng production costs adjusted'to reflect a production entity of one. test a*l evaluation vosts of -search pr .gra» atr no' tm Uded. 

see discussion In Chapter 111 and Table IS. 
c lMs Is a n«xt«M value. Where data did not provide a separation between recurring and ngn recurring production .y,f fh<- *Jlu* *Mi«n 1s *ot»» 

production cost Where data did v»t allow an estltiat* of production cost no valur Is shown 
^Source Progra« Of f ke • 
*See lable HI 

— f $oyrc(r joint Ucttcal electronic Designation System. Master consolidated Reference List. CKtoc-r \W\ tonsu I 1dat#*d M«naj«»er.« Ua'a 
October 1980 

9 CICChlnel1i. <t 1980. P 68 The actual t,jul[*>ent trainer is an operational 6883 test b$*>ch with *n nodif lcat ions The value rt 
H.9S5.00O represents the cost of the operattonal test bench In adjusted for »ubse*3uei't piue level lhan.jrs. »t e«< Tides per en 

attributed to acquisition -runagi^ent 

**10 studerr stations 
\ 



Per Student station 0 _ 

second unit is currently be i n g~ if i~F> ' i cated See^ lab I e 1 2 
k f«Ch unit consists of 1? Student stations, each student station is outfitted with \2 different simulates ( 
WtCC-H. AN/IRC- 1 ». AN/W-H5. AX/TSQ-cU. Afi/WBS 

m Q ne network con sisting of foyrsifr-ulateij stations that can operate independent \y or in nn» eH 

n Progran cost is in dispute The -contractor has filed claim for an addlt.pnal su llion 
%|ulpn«nt required for one class of IS students 

p ftve co*pl**es of six trainers each All completes provide operator training, two corrlU.es provide both . peratoi and f*an»enan e -ra r>i»g. 

(one for the AN/TPQ-36 and one for the AN/TPQ V) 
q (ach trainer contains eight different simulations (panels) 
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(before modification for use in training) in 20 maintenance 
simulator programs, it is important to understand the different 
t^pes of cost data ^hown ,in this Table- 
V Operational Equipment Unit Cost - Production cost of an 
additional unit of equipment designed to meet some 
military purpose; these values do not include RDT&E 
costs. Where the costs of an actual equipment trainer 
are not available, these costs a~e used as a proxy 
for actual equipment cost. 
Actual Equipment Trainer Unit Cost - Cost of operational 
equipment, immediately above, that has been adapted 

for use in training, e.g., power,_ s^cj^l^inputs-^-' ; 

coatroUr- et-c; — Saeh^rToidirfTcations require additional 

costs. 

^Simulator Total Program Cost - Cost'oT RDT&E, prototypes, 
and manufacturing facilities needed to produce one 
or more maintenance simulators. In our sample of 20 
simulator programs for which total program cost data 
could be compiled, there were 12 instances in which 
only one simulator was built; in the eigjitjother 

programs, f-rom-2 to 3 6. units were built. 

Simulator Normalized Total Program Cost - Totai production 
costs adjusted to reflect a production quantity of one; 
includes the costs of research and development but not 
test and evaluation of simulators developed in research 
programs . 

Simulator Unit Recurring Fabrication Cost - The^ost of 
producing a folic '-on unit of equipment after the 
costs of RDT&E, prototypes, and manufacturing facil- 
ities have been accounted for. This is a maximum 
value; where data did not provide a separation between 
recurring and non-recurring production costs, the • 
value shown is total production cost. No value \s 
shown where cost data did not allow an estimate of 
production post." / 
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There are, thus, several ways to compare the costs of /ac- 
quiring maintenance simulators and actual equipment trainers. 
The cost of an actual equipment trainer is set, approximately, 
by the incremental cost of procuring one additional unit of 
operational equipment plus the cost of any modification neces- 
4 sary Cor its use in a classropm. Thfs is a unit recurring 
fabrication cost, devoid of RDt&E and non-recurring production 
costs. We have these costs for six.AETs. Where AETs have not 
been built, we can use the operational equipment unit cost as 
a „s uc r aqa tc AET cost for- comparing- the costs of simulators and 

AETs . ' 

c The average ratio of AET unit cost: operational equipment 
unit cost is 1.27; that is, AETs cost, on the average, about 

^ percent more than operational equipment before the latter 

is modified for training (Table 17); the data are based on only 
. five cases. These ratios, which vary from 1.00 to 1.59, presum- 
ably relate to the degree of modification involved in the vari- 
ous cases; whether further modification at even greater cost * 

would improve the effectiveness of instruction has not been 

examined . 

Wo will estimate the cost of acquiring a maintenance simu- 
lator in two ways. The first estimate includes non-recurring 
cos'ts (e.g., research, development, and manufacturing facilit os) 
and the costs of manufacturing one unit. This value is the nor- 
malized total program cost, as defined above. The second esti- 
mate includes only'the unit recurring fabrication cost (as de- 
fined above), i.e, the cost to produce an additional unit 
(after research, development, and other non-recurring functions 
have been accomplished). Each of these estimates is relevant be- 
cause of the large disparity between recurring and non-recurring 
costs. Maintenance simulator programs have typically involved 
small quantities so that relative cost-effectiveness of their 
use will vary qreatly with quantity procured. To the extent 
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TABLE 17. COMPARISON OF THE ACQUISITION COSTS OF 
ACTUAL EQUIPMENT TRAINERS AND COMPARABLE OPERATIONAL EQUIPMENT 
BEFORE IT WAS CONFIGURED FOR TRAINING 





Unit Cost 
(thousands of dollars) 






Operational 
Equipment 


Actual 
Equipment 
Trainer 


v Cost Ratio': 
AET/Operational 

Fflij i nnpn t 


~MA-3 " 


no 


175 


1.59 


Hfriderrt^lnreg rTted 
Radio Room 


i ? i nn 


1 7 ^nn 


1.45 


Trident High Pressure 

«ir uuniprcoour 


..315 ... 


.400 


1.27 


Trident Air Conditioner 


530 


550 


1.04 


F-111 Avionics Test 
Bench (6883 Test^Stand) 


1,955 


. 1,955 


1.00 
Mean 1.27 











permitted by the data, we have estimated the recurrinq costs 
even if only one unit was actually fabricated. 

The actual equipment trainers and operational equipment 
shown in Table 16 vary widely in cost (from $45,000 to 
$17,500,000). Therefore, we have used ratios to compare the 
costs of simulators and actual equipment trainers. The central 
tendencies ot the cost ratios, for both the normalized total 
program and unit recurrinq estimates, are shown at the bottom 
of Table 16. 

Plots of the individual ratios of estimated simulator 
costs (both recurrinq fabrication and normalized proqram) to 
actual equipment trainer and operational equipment costs are 
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shown in Figs. 6 and 7.* In both figures the operational equip- 
ment costs have beer adjusted by the average ratio of costs de- 
veloped in Table 17. With two exceptions, the recurring fabri- 
cation costs of sfmulators j(Fig. 6) are 20 percent or less of 
the costs of either operational equipment (as adjusted) or 
actual equipment trainer?, and this conclusion does not^ depend 
upon including operational equipment in the sample. Nine of 
the _11 cases (80 percenc) fall below this. arbi trary threshold, 

but there is a large dispersion among them ranging from 3 to 

19 percent. Available data ^provide no explanation fpr this 
range. The available data provide some insight int^ the two 
cases that fall above 2 0 percent. The VTAS simulates avionics 
equipment that has been out of procurement for many years, and 
we\ suspect that the cost of 'the operational equipment ls-seri- 
ousl'y underestimated. . The ma/ 3 is a research device and may 
contain features that seVve only the research function. How- 
ever, it does not appear , reasonable th&t these special features 
.alone would account for its relatively hiqh»jcost. 

The relationship between simulator normalized program costs 
and the costs of actual equipment traine/s or operational equip- 
ment are not as clear-cut (Fig. 7). In seven of the ll cases 
the simulates cost is 'less than 60 percent of the cost of the 
actual equipment trainer r operational equipment (with a ranqe 
of 2 r > bo 55 percent). Howover, in the other four cases, the ■ 
percentages jrange from 160 to 400: At first appearance, this 
sample seems to come from two populations, but we can find no 
support for this argument in the characteristics of either the 
simulators or the procurement programs. Similar to the prevfous 
ratios o!> recurrinq (fabrication) to actual equipment costs, 



-*The Trident I RR maintenance trainer has been excluded from 
this analysis as it appears to be as much a complement to as 
a substitute for either the act ual -equipment trainer or the 
operations/maintenance trainer. t • 
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FiGURI 6. Relation between actual equipment trainer and 
simulator recurring fabrication cjosts 
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the VTAS and*MA-3 simulators are "outliers" here too, and we 
suspect for the same reasons. The AT Trainer also simulates 
avionics equipment that has been out of jproduction for a number 
of years, and we^ suspect the^ cost of the operational equipment 
is considerably underestimated. We have no* explanation for the 
relatively high cost of the AWACS Navigation/Guidance Simulator. 
The contractor of this program incurred a significant (and non- 
reimbursed) overrun that has been attributed to his independent 
development of a courseware translation system (discussed in 
Appendix, B). Ewsn when, the overrun is subtracted, the normal- 
ized program cosJt-still exceeds the full adjusted cost of the 
operational equipment by close to 25 percent. _ - 
» Note that three of the four cases with relatively high 
cost ratios^ involve the comparisons with adjusted operational 
equipment costs, and again, it appears that the sample comes 
from two populations. For example, in 8 0 percent of the cases 
where ratios "based on actual equipment trainers, the simu- 
lat ^P? brmalize ^ P ro <3 ram cost- is less v thah 50 percent of the 
unit cost 6f>he actual equipment trainer; for those ratios 

&-QA^4«stod^^ 

program cost is greater than 50 percent of the unit operational 
equipment cost in 8 0 percent of the*cases. We can firia.no « 
rationale tor this observation. No such distinction *can be 
made with respect to simulator refcurring fabrication cost, and 
w£ feel it, is spurious; * ** 

The cost-effectiveness of a maintenance simulator on a 
l^fe-cycle basis has been evaluated only intone case, that of 
the Air; Force 6883 Test Stand 3-d imensional simulator and actual 
equipment trainer (Cicchirielli , Harmon, Keller,/ et al., 1980). 
In a later sttady, thes^ authors will also evaluate a ,2-d imen- 
sional versipn of this simulator. The 3-d^imenSional simulator 
and actual equipment trainer were equally effective wheri meas- 
ured- by student achievement at school; 'supervisors' ratings 
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showed no difference between the job -performance of students 
trained either way for periods up to 3 2 weeks after leaving 
school.' . ^ ~~ " ~ " ~ > 

The life-cycle cost comparison gf^simulator and actual 
equipfoerft trainer is shown in Table 18. Costs were estimated 
in constant 1978 dollars over a 15-year period ahd discounted 
at 10 percent. The results .show that the cost per student hour 
was $23 for the simulator and for the actual, equipment 
trainer, i.e. , 38 percent as much for the simulator, compared 
to the actual equipment trainer, for all costs over a 15-year 
period. . The simulator cost less to procure ($594,000 v^s 
,$2,10$, 000, -or 28 percent as mu,ch) and less to operate 

($r*,588,000 vs $3,367,000 or 47 percent as much) over a 15-year' 

* * * 

period. * f * 

, ^ Using net present value (1978 constant dollars), the recur- 
ring costs were $1,791,000 or 8 5 percent of the non-recurring 
costs of the actual equipment trainer. The recurring costs of 
the simulator were $906,000 or 152 percent of its non-recurring 
costs., « * < I 1 

We draw the following conclusions: 

Cotst : Maintenance simulators cost less than actual equip- 
ment trainers. On the average, to develop, and fabricate 
one simulator costs less than 60 percent ofN^ie cost of 
an actual equipment trainer; to fabricate one unit of a ' 
simulator Jonce it has been developed ) , costs less. than 20 
percent of the cost of an actual equipment trainer. How- 
ever, there is a large dispersion about these averages. 
Effectiveness i Achievement* at school is ttfe same whether 
students are trained, with maintenance simulators dr with 
actual equipment trainers. This finding applies to 12 but 
of 13 case£ in which such comparisons were made. There- 
t fore, maintenance simulators are cost-effective compared 
*to actual equipment trainers. * ' 
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TABLE' 18. LIFE-CYCL£ COST COMPARISON OF 6883 TEST STAND, 
ACTUAL EQUIPMENT TRAINER AND 3- 0 1 MENS I ON AL SIMULATOR 3 





Costs 

" (thousands of dollars) 


Cost Category 


Actual Equipment Trainer ' 


3-Dimensional 


• Non- 
recurring 
'Costs 


Recurring 
Costs 


c No n — *• 
4 recurring 
* Costs 


Recurring, 
Costs 


Facilities 


1 


110 


■ i • 


no " 


Equipment 


2104 


2798 


.594 


1000 


Instruction*! 
materials 




28 




to r 


Instructors and 
overhead 




73 


• 


QA 


Students and 
support 




358 




a 

358 . 


Total 
Grand Total 


2105 


3367 
5472 


*59? 


1588 
« 2183 


(Net present 
. value, 1978) 




(3896) 




(1501) 


Cost per student 
hourb 




. $60 




.$23 



Estimated based on 15-year life cycle discounted at .10 percent, in .1978 
constant dollars. Modified from data presented-in CiCchinell 1 , Harmon, 
Keller, et al., 1980, p.* 67-69. Table corrected to show cost of instruc- 
tors for simulator and cost per student hour over a 15-year period for 
AET and simulator, based on discussion x with senior author. Analysis 
assumes 720 instructor hours per year an^operation of equipment for 
2.1 shifts per day to handle student load. N J 
/ * . ' ' . 

180 students per yr x,3 days «per student x 8 student hrs per day x 1*5 
yrs = 64,800 total student houcs. * . 



This finding is necessarily, qualif ied'by the limited nature,^ 
of the data from which it is derived. Effectiveness, as^used 
here, is based on performance demonstrated at school rather, than 
on the job. Cost, as used here, refers to the initial cos-ts *of 
acquiring training equipment and does not incljude the costs 
associated with the operation of simulators or of actual equip- 
ment for training, e.g., maintenance and upkeep, instructors aftd 
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support personnel, student time, and the like. In the one case x 
where a life-cycle cost comparison was made, total costs over a 
15-year period for * the 6883 Test Stand' 3-dimensional simulator 
was 38"ptercent*as much as fot the actual equipment trainer. 

Both were equally effective, as measured by tests at school and 

*** " - I 

by supervisors 1 ratings'^ on th^ job after school. 

^ Insufficient information is available with which to draw 
conclusions as«to whether 2-D simulators are cost-effective com- 
pared with 3-D simulators, the aspects of maintenance training 
for which simulators are most effective, and how to allocate 
thevamount of time, for greatest cost-ef fectivjeness, between 
maintenance ^simulators,, actual equipment trainers, and on-the- 



> job training.. All of these topics are matters for further 
.research, development, test, and evaluation that are discussed 
next in this paper. * 
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V. DISCUSSION 



On the basis of data presented in previous chapters, we 
find fh at m a in fcgn^n^ "s imulators are cost-effective compa red 



>ith actual equipment trainers. Both are about equally effec- 
tive for training maintenance technicians at schools; in general, 
maintenance simulators cost less to acquire than do. actual equip- 
ment trainer^. In this chapter, we wish to discuss the signi- 
ficance of these findings ^ the limitations of the data upon 
which these findings are based, and the steps that should be 
taken both the improve our knowledge and to increase the cost- 
ef€ectiyeness of maintenance simulators^ used to support train- 
ing .for future systems. 



A. EFFECTIVENESS / 

Students trained on maintenance ^simulators perform as well 
on tests at school as do students trained on actual equipment. 
This finding is consistent with results of studies with use of . 
computer-based instruction for technical courses on electricity, 
electronics, vehicle repair, precision measuring- equipment, and 
weapons mechanic (i.e. .nob maintenance training per se). 

We would expect that individualized, self-paced instruc- 
tion, an inherent characteristic of maintenance simulation „, 
would save some of the time students need to complete instruc- 
tion given with equipment, particularly where the actual equip- 
ment trainers are used more for classroom demonstration than for 
individual practice by students. Only three s^udies^ of mainte- 
nance simulators report data on the time needed by students to 
complete their courses (Parker and De Pauli 1967; Rigney, Ttowne, 
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King,/et al., 1978; and Swezey 1978).* Here, students saved 22, 
50, and 50 percent, respectively, of the time needed previously 
in bourses given with actual equipment trainers. Time savings* 
if any, when maintenance simulators are evaluated, should be 
recorded in future studies; it is anticipated that the few re- 
sults reported so far will be confirmed. It is important to 
understand that all findings, although positive with respect 
both to simulators and computer-based instruction, apply only 
to training at school; there are no data about the effect that 
such training, including that using actual equipment, may have 
on job performance in the field. The importance of collecting 
information about maintenance performance on the job, and relat- 
ing it to method of training at school," can hardly be overempha- 
sized. • 



B . COST. " 

-Maintenance simulators appear to cost less . to procure than 
do actual equipment trainers* With some exceptions, the cost to 
develop and fabricate one simulator i$ less than 60 percent of ' 
the cost of an actual equipment trainer, "and to fabricate one 
unit of a developed simulator is less than 20 percent of the 
cost of an actual equipment trainer. This finding is based on 
11 cases where meaningful cost comparisons could be devised., 
Because of the limited number of cases, no attempt was made to 
investigate the determinants of cost. t * 

It is important to emphasize that these comparisons are 
based only on procurement costs; they are not life-cycle costs. 
Simulators and actual equipment trainers are used for training 
over relatively long periods of time. such as, for example, 10 
years. In addition to the costs of acquisition, they incur costs 
for operators, maintenance, instructors, and students. There- 
fore, life-cycle costs are more significant than acquisition 
costs alone* as a .basis for evaluating the costs of alternative 
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training devices. There is one recent estimate that the life- 
cycle* costs of a training program using a maintenance simulator 
(the 6883 Test Station) would be about 40 percent that of one 
using the actual equipment { Cicchinelli Harmon, Keller, et .al. , 
1980). " . ' — 

However, the cost data that are„ now available are not ade- 
quate for definitive conclusions regarding' the cost-effective- 
ness of simulators vis-a-vis actual equipment trainers. ; Some 
of the data that are available appear to ^ogtain such systematic 

biases .that their reliability may be questioned- The way. 4n 

which maintenance simulators are" procured appears to contribute 
to the inadequacy of currently available cost data; 

• The cost of simulator programs fall below the cost 
threshold of major procurements^^tfith their associate 
requirements for use of a standard work breakdown 
structure (WBS) and-forr-contractor cost reporting 
within the WBS structure. Contract line item listings, 
that might sierve-as a functional cost structure, vary 
considerably, both among the Services and among separate 
contracts within a single Service, with a result that 
cost documentation may not be comparable among contracts. 

• Most maintenance simulators wittf standardized software 
systems have been procured by t means of firm-fixed-price 

,(FFP) contracts. Here, the only costs that are general/^ 
available are limited to those spelled out in the con- 
tract itself. For the systems with non-standard' soft-/ j 
ware, f ixed-price-incentive-f ee (FPIF), and cost-plus/ 
.fixed fee (CpFF) contracts, and CPIF have also been / / 
^employed. However, FPIF contracts provide .the Services 
* with little leverage in requiring contractors to pr 
v\de cost information; none of the program offices/that 
have employed cost-plus contractors have required /con- 
, tractors to provide this type of data. 
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Some of the maintenance simulators for which cost data 
are available were procured "primarily for purposes of 
research arid development* In these cases, both the 
programs and the resulting devices contain features that 
would not be present had tl^e devices been intended only 
fd"r routine training* Costs for features peculiar to 
research may be considerable but, in general, they can- 
not be identified and Separated* Thus, such cost data 
probably contain an upward bias* 7 

We believe, but cannot document, that currently 
available cost data on maintenance simulators must be _ _ 
qualified even furthed for the following reasons: 

!• Contractors of some systems with non-standard 
*. software appear to have incurred losses that 
(in whole or in part) they have not divulged* 
This would introduce a downward bias in the 
available data* Thtfs judgement has been offered 
by personnel in the program offices involved* 
2* Contracts for standardized systems typically 
encompass procurement of several different 
simulators; negotiations appear to focus on 
total contract cost* This has two impacts: 
o (l) it 3*1 lows trade-off$ among individual con- 
tract line-item costs in order, that the rela- 
tionships among them appear •'reasonable 1, to the 
government; (2) contractors have an incentive 
to inflate Jthe costs qf simpler devices and de- 
flate the costs of more complex devices* ' The 
result is to distort the cost relationships 

1 

I among contract- elements* , 

,3* The market appears highly competitive for sim- 
ulators with both standard and non-standard 
software, and it is difficult to get contrac- 
tors to provide detailed cost da.La* 

, 4 - 4 
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4. Within the small sample of ribn-standard systems, 
there exists a wide range oqf program^ arrange- 
ments, device complexity, physical characteris- 
tics*, and training capabili/lties. Considering" 
this Wide spectrum, it is Questionable whether 

- % the sample provides a sufficient fease for devel- 
oping a cost analysis capability, even in the 



absence of the cost data 
above. 



problems discussed 



JCQS!T-E f f e cti ven es s 



Maintenance simulators appear to be .cost-ef fecti\je compared . 
td actual equipment trainers for training technicians/ 0 Since 
the qualifications that apply to i:his finding "have been 'explained 
above, they will only be^cited here:, the. finding is* based on 
-acquisition rather than life-cycle costs, on effectiveness as 
measured by the performance of students at school rather than 
on- the joband only on a limited number of cases (N=ll), Here, 
we will tjry to explain what this finding does and does not tell 

• US# • - • / ' 

We can realize the cost advantages of maintenance simula- 
tors only by using thejn instead of actual equipment trainers 
(aptong. other training resources) An our maintenance training 
programs. This is likely to cause some problems for those who 
believe that, even if maintenance simulators are used, it is 
"still necessary to use actual equipment trainers at school to 
train, technicians how to work/'later on actual equipment on -the 

job. This dilemma can be resolved by comparing on-the-job ppr- 

'* It ■ 

formance of those trained at/I school only with different mixes 

of both actual equipment trainers and simulators. An evaluation 

> y // * » ' 

of ^on-the-job performance Has been reported in only one instance 

(Cicchinelli et al. 1980). j\ Here, supervisors 1 ratings showe,d 

^tftf~#llf f erence in the job performance *of technicians trained 
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bnly wit^ the 6883 Converter/Plight Control System l4st Station 
3 -dimensional Simulator or with thfe actual equipment, trainer . 
Additional studies of this type would be most welcome. 

Sijt>pos* that -on-the-job performance tu£ns out/ to be the 
same f6r stud^pts trained only with simulators orwith a cfcm- 
binW:pn of simulators and actual equipment thrainers.* If stu- 
de ^ t ?' 1 loads were such that only oneitfcm of training equipment 
were/ocequired, then the additional costs attributed to the 
act/ial equipment trainer would make the combination more costly 
ar}4 no more effective than using only the simulator for train- 
ing* If; simulators cost less, the same resqlt would also apply 
t'$ cases* where, because of a large Student lc/ad, two or more 
jf terns of training ^equipment were required* ; The school might 
^Gse some , combination of 0 simulators and actual equipment traine 
|This type 6f compromise, while not most cost-effective, might 
[appear reasonable to skeptics who believe that actual equipme 
traine 3 are still required. Since, except for Cicchinelli, 
Harmon, Keller, et al., 19&0, there are no data to support a 
'reject the ? notion' that both actual equipment t trainers and simu- 
lators ar^( needed for adequate training, /there is much to be 
g.ained by. Collecting the job per c ormancef data needed to resolve 
this dilemma. / ' j 

Maintenance simulators, it h^s be^?n ai „*'ed, cost less and 
are more effective than actual, equipment trainers because they 
provide feedback to students, provide /training * in a larger num- 
ber of malfunctions than is otherwise ? /possible, and have fewer 
breakdowns when 6sed by students. cJst data support the first 
claim;" although the other claims appear plausible, there are no 
data 'to support (or reject) any°of them. Some enterprising ' mili- 
tary laboratory is invited' to consider these questions. 

Maintenance simulators provide individualized, self-paced 
oinstruction 4 and, because of t&is, l>ne would, expect them to save 
some of the time needed by students to complete the course of 
-instruction. This result has, in/ fact, been reported in three 



studies. If confirmed, as we would expect,, the cost avoidance 
.attributable to reduced expense for students* pay and allowances 
at school would increase the cost-advantages of simulators/ 
This type pf calculation has- not been included in any evaluation 
of maintenance Simulators. * , 

It", is conceivable .that some mainteriance simulators would 
be more* expensive tio, procure than actual equipment trainers for* 
the sarne^ applications. If. all other't^pgs, e.g., effectiveness, 
are equal, then we should obviously choose, the less expensive 
option.. However, "all other things" are rarely equal. A simu- 
lator, though morq expensive" to procure th^ja an actual equipment 
trainer, might sufficiently shorten student ^time at school, re- [ 
educing the need- for instructors and support personnel; to be 
le^s expensive on a lif e-cyole-cost basis; it might also improve 
student on-the-job performance sufficiently to be cost-ef receive 
in terms o,f the combined costs of training and (subsequent) 
maintenance. This statement is not intended to be an argument 
in favor of simulators. Rather, it is made to point out that,^ 
up to now/ all studies .of simulators and actual equipment 
trainers have been One-po/nt comparisons, i.e., equal effec- 
tiveness and lower costs < *?or simulators. Since no studies have „ 
been made between training devices of differing levels of both * 
tost and effectiveness and that extend the analysis to later" 
performance on the job, #fc is not yet possible to look for an 

optimum combination of maintenance training equipment. 

» * • 

s 

D. RISK OF CONCURRENT DEVELOPMENT 

There is an .obvious advantage in the concurrent development 
of operational and training equipments jso that trainers are 
available when, and preferably before, the operational equipment 
is first fielded. However, this practice also entails risk, 
since even minor changes to ? he operational equipment may result 
in latge additional costs to modify the .training simulators- 
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(especially in the areas o£*simulation*sof tware and courseware) 
In five cases that havy been identified as concurrent develop-* 
pments, ^significant portion of final simulator costs has been 
attributed to modifications in the operational equipment. « 
Although the sample is small, it suggests that concurrency will 
increase the costs of simulator dc<v< lopment progtams< 



It also follows that if Simula 



:ors fop training are de- 



veloped only after the design of thi^ofJeratipnal equipment 'has 
been frozen, t^e simulators may not] be ready for training when 
needed* A possible alternative is ito. train the initial cadre 
of personnel' with actual equipment /or with simulators based on 
a preliminary design, knowing » that/ more adequate simulators, 
will be built later. Whether or dot such an. alternative is' 
both effective enough and not too/costly is a topic for system- 
atic study.* n * 



/ 



E. IMPORTANCE* OF SOFTWARE COSTi 



Software and courseware (iJ<*., program design and pro- 
gramming) appear to be major elk^nts of cost in non-standard, 
maintenance simulator systems. ' No hard d^t-a were foilnd on this 
point; nevertheless, it is the/opinion of individuals" who have 
been involved with the managentent of maintenance simulator pro- 
grams. According %o these individuals, cost overruns that have 
occurred have been, due primarily "to profile j in developing 
software programs. Should this be true, it points to a cost 
•advantage to be gained fcy deyfcloping widely applicable soft- 
ware systems fop the more complex training applications. 
Although no data are available on this point, this would not 
apply to standard systems, ^ince thfe same software system i% 
employed in all applications/ developed by' one contractor. 
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P. LIMITED POTENTIAL FOR QUANTITY PRODUCTION 

* , «-« — 

The bulk of potential maintenance simulator 'applications, , 
appears to arise in system-specific training (as opposed to 
general skill training), and a majority of the more promising 
candidate applications seem Jto be associated with aviation 
training. However, training for a specific model of aircraft^ 

is concentrated at a small number of sites and involves low . " 

^ ~*» 

rates of student flow. As a result ,* there is a limited poten- 

*> * 

tial for quantity production of a given model of simulator 
oV.er which development costs can be amortized and a limited 
opportunity for reducing unit costs through a -widespread adop- 
tion of maintenance simulators. 

G. What should be done next 

1. Cost-Effectiveness Trade-Off Studies 

Maintenance simulators have been found to be cost-effective 
although the data for this finding are limited. There is no. 
reason to Soubt the same" result for additional comparisons of 
maintenance simulators and actual equipment trainers. Neverthe- 
less, we^ Y should kn^w how to optimize the design and use of main- 
tenance simulators and to be, able to make trade-offs between 
their effectiveness and cost. There is almost a total lack of 
systematic knowle<Jg^about the relation (i.e., trade-offs) 
between effectiveness and cost in the design and use of mainte- 
nance simulators; for example, what features increase their 
eJLflectiveness in particular applications; conversely, , 1 ittle, 
is known regardi#$f the relationships between simulator features 
and their costs. Simulators can, naturally, increase in cost 
in many ways, such as by including more malfunctions in their ' 
courseware programs, by providing more complete realism in 
appearance and functional capabilities (in both*3-D and 2-6 
designs), and by providing more computer-based, instructional 
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.guidance" to students? _the converse of any of these statements 
may-a^so be considered. A substantive question is to deter- 
mine the extent to which increases in the capability of mainte- 
nance simulators (with associated increases in cost) improve 
the , effectiveness of training, i.e., student performance, 
beyond 'that which can be achieved without these incremental 
costs. \No studies have been undertaken to explore such ftinc- 
•tional Relationships, except forHhe still-to-be-completed 
vjiM 0 ^ o^f Cicchinelli, Harmon, Keller, et al w 1980 that will 
compare 2-D and 3-D versions of the 6883 Test Stand. 

2. Validate Simulators with Performance on the Job 

Student performance at school is, at best,^ an indirect 
measure f<|r evaluating the benefits of simulators, compared to 
"actual equipment trainers, at schools. The real issue is to 
compare how training with either of these devices improves the 
ability of bourse graduates to maintain "equipment on the job. 
The purpose of> school training is to gualify students to per- 
form well on jobs in the field and not, per se, to complete 
a course at school. Dc:a to show the effectiveness .of main- 
tenance simulators, compared to actual equipment trainers, as 
measured* by field performance, is totally lacking and is essen- 
tial for definitive evaluations. Cicchinelii, Harmon, Keller, 
et al # , 1980,- an oft-rited reference in this paper, reported 
supervisors' ratings of on-the-job performance of course grad- 
uate^ but did not collect objective data on the actual perfor- 
mance of these individuals. 

3 ♦ Fidelity of Simulation * , 

Instructors, in general, favor the use of actual equip- 
ment, rather than simulators, for the training of maintenance 
personnel. Reasons given for this preference are that students 
need to train with actual equipment and that the lack of real- 
ism in simulators can interfere with effective training. Such 

* 
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reasons cannot be dismissed, because the views of instructors 
;can influence the way in which simulators arexised in a course; 
an inappropriate use of a simulator may easily make it not inef- 
fective and therefore not efficient for training. 

A number of ^studies have shown that low-cost devices, such 
as mock-ups, charts, and flat panel simulators are as effective* 
as real equipment for training personnel to operate (rather than 
maintain) equipment (Grimsley 1969; Denenberg 1954; Torkelson " 
1954; Swanson 1954; Vris 1955; Spangenberg 1974; French and 
Martin 1957; Prophet and Boyd 1970, Dougherty, Houston, and 
Nicklas 1957; and Cox, Wood, Boren et al., 1965. Useful reviews 
of this topic may be fpund in Micheli 1972, Kinkade and Wheaton 
1972, and Fink and Shriver 1978). These studies show that stu- 
dent achievement (i.e., learning the required information) is 
about the same with real equipment, expensive simulators, or 
-inexpensive mockups,; this is taken to represent arrange of high 
to low fidelity in these devices. Some studies have shown that 
there no differences between individuals trained on high or low 
fidelity devices when measured by training time, amount, of in- 
formation remembered (after .4 or 6 weeks), or time devoted to 
additional training some time after leaving school. These 
findings apply primarily .to teaching procedural tasks \ e.g., 
.nomenclature, equipment start-up, malfunction location, and 
troubleshooting logic. This evidence cannot be denied, but; it 
has not had a major influence on the design or procurement .of 
maintenance simulators. 

, All recent studies hf maintenance simulators, have eyalaated 
a specific simulator as a direct alternative to some actual 
equipment for training purposes. Fidelity was not varied* sys- 
temically or otherwise in any of these studies, with one excep- 
tion. Flat panel (2-dimensional) and 3-dimensional versions of 
the 6883 Test Station simulator were developed so that a direct 
comparison could be made of their effectiveness for training 
maintenance technicians. The three-dimensional version, produced 
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by Honeywell, has been evaluated for use in training technicians 
to operate and maintain "the 6883 Test Station (Cicchinelli , 

Harmon ; TCeller , ' ef al. , 1980). The two-dimensional version, 

produced by Burtek, will also be evaluated, but data collection 
and evaluation had not been completed when this report was ° 
written (April 1981). 

There has been no effort to interpret what aspects of „ fi- 
delity, if any, may have been varied in the studies that have 
been completed to date, although .it is obvious rhat different 
pieces of* equipment with different methods of presentation 
were* involved and that these devices cost different amounts of 
money. There is, at present, no way of measuring, scaling, or 
defining what* we mean by the fidelity of a training device, 
particularly with respect to its effectiveness for training* 
students. A distinction made by Milled (1954) between psycho- 
logical simulation and engineering simulation (and copied 
assiduously ever since) does not help very much: " engineering 
simulation- Jisl the copying of some physical model and its 
physical properties" (p ? 19; emphasisT in original ); -psycholog- 
ical simulation.... provided stimuli so that responses learned 
to them will transfer from training [with training devicesj to 
operations with little or no loss" (p. 19 ). ^ "Psychological 
simulation may be far removed from physical realism" (p. 20). - t 
"The development of training devices should rest on psychologi- 
cal simulation rather than engineering simulation" (p. 20'). 

It may be that less expensive devices are as effective, 
as more /expensive ones for maintenance training. However, we 
lack both a metric and a guideline to identify either the 
physical or functional characteristics of these devices that 
influence the effectiveness of training. The interrelation- 
ships of complexity, fidelity, and cost 'of straining ecfffioment 
and the transfer of training from training devices to oper- 
ational equipment clearly deserve systematic attention, both 
for R&D on training devices in general and £or particular 
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emphasis on maintenance training. Different level* of com- 
plexity and of 'fidelity may be\equired for manual, iWls-on 
skills needed in maintenance than for those which involve pri- 
marily knowledge and procedures associated with diagnosis of 
• malfunctions and troubleshooting. 

4. On-the-job Training - * 
Technical traihing^at school 'qualifies. a maintenance .tech 
mcian to undertake further training on the job and is not 
. expected, by itself, to produce' a high level of competence. 
At stake, therefore, is assessing the cost-effective mix of 
training at school and on the job^ This important question is 
beyond the scope of this paper. Nevertheltss, a potential", 
advantage of maintenance simulators, particularly as the newer 
ones become more compact and portable, is that they would per-, 
mit. us to measure the performance of maintenance personnel on 
or near their job sites and, where deficiencies are found, to 
provide refresher training to particular individuals. Thus, 
maintenance simulators provide a means of collecting objective 
data. about technicians oh the job (in a test-like situation), 
that could be used to validate not only the use of simulators 
in school but of actual equipment trainers; this also applies 
to any-other feature of interest in the type of instruction 
offered at schoo.l. 

5. - Research on Maintenance 

, /E veh after about 3CT years of research on maintenance 
training, we still lack some fundamental information about how 
humans perform the task of r.ainfenance. As a consequence, 'we 
cannbt accurately specify, "-as suggested above, whether a partic 
ular. simulator. sftould he simple or complex, two- or three- 
dimensional . x n form, the' optimum 'mix of general and specific 
maintenance training, and 'the .trade-off s between increased re- 
liance..^ automatic and built-in test equipment versus reliance 



on human ability tp diagnose and perform various maintenance 
procedures. "At least in principle, it is feasible to improve 
built-in test- equipment t_o assist the 'technician to find mal- 
functions # and to design a .system so that components and t*st 
points are more accessible to maintenance personnel. The real 
issue is to determine whether increased expenditures during 
system development for engineering characteristics to facilitate 
maintenance wi.ll reduce expenditures for. personnel, training, 
maintenance, test equipment, and spare parts over the life^ 

cycle" -of that system-. . . . , ' V " 

It' is not yet possible to measure ^complexity of main- 
tenance 'tasks so that specifications for equipment which have 
an impact on maintenance and maiatenance personnel can be set 
•both for the design of weapons systems and for maintenance . 
simulators and training programs .(see Wohl 1980; Rouse; Rouse, 
Hunt, et X al., 1980; Nauta'and Bragg 1980). 

It i's not. yet clear to what extent maintenance simulators 
should be designed to^provide jgeneric training applicable to a 
variety of equipments and/or .specific training applicable primar- 
ily to particular/models of- equipment". A current program at the 
Navy Personnel Research and Development Center is attempting to 
address this issue (the Generalized.Maintenance Trainer System). 

• There, are insufficient data on the amount of time re- 
quired to. find , identify, and fix various types of malfunctions. 
Without such data, there appears to be little rational\basis ,f or 
determining the priority to be given to various types of mainte- 
nance tasks included in maintenance training course and, of 
course, in the design.pf the maintenance simulators' .to be used * 
.in these courses (Johnson and Reel 1973). .The work o£ Rouse, 
•Rouse. Hunt, et al. ( 1980) suggests that .the more difficult,, 
fault isolation tasks-are in equipment with feedback loops; 
1 humans benefit during training when t'hey are given immediate 
^knowledge of results about the rules they are using. to identify 
faults; these'" skills appear to be transferable to situations . 
where immediate knowledge of results is' not provide^ 
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V V 



simulator 



COURSE 



COURSE 
LENGTH 
(STANDARD) 



EOUniEHT 



Generalized Sonar in 
Maintenance Trainer v ; * ' 
(GSMT) * / 



Sonar. maintenance 
(Special course for 
this experiment) 

Fleet Sonar School 
San Olego, CA - 



4 days 



\ 



AET: AN/SQ-4 Sonar 



SIM: GSMT 



^Intermediate general 
electronics (4-week 
segment of 14-week 
course In^onar Main- 
tenance Training) 

Fleet Sonar -School 
'Key West, FL 



Hralns calibration alignment, preventive 
maintenance and troubleshooting of cir- 
cuits and components common to six 
sonar systems." 

(2^ * ' - • £ 

y /Control group 1 tralned'before atfd control 
group 2 trained after experimental group. 

^Percent correct answers to special test 
with 141 items. - 



(4f 



This is a low- power sonar system rather 
than a true simulator. 



4 weeks 



Note: Final test for 
both groups on* 
AN/SQS-23 sonar 



AET 



SIM: GSMT 




J30. 



w : ■ 


* t# . 


-* 

»"*■'_ v 


- \ 






■ 


. COMPARISON 


• 


\ 




; HO. OF 
STU06NTS 


SCORES 


TIME 


ATTITUOES TOWARDS , 
SIMULATORS 


cwM^Eias 


REFERENCES 


^- 9 


Accuracy: * 
Same or better- 


Average 22 percent 
faster in perform- 
ing maintenance 
tasks on test * 


Students favorable 

'** — 


Transfer of # jtra1n1n<K 
experiments StudentsX, 
trained on AET or simu- \. 
lator; both groups ^ 
tested on a new sonar. 
Performance compared on 
five maintenance' tasks. 


Parker and DePaulTr - ' 
1967 _ j 

\ 1 

\ ' • -1 




Criterion Avg. of 
TestC 3 ^ weekly 
tests 






7 S 1 mu 1 a tcfrnjrou p~ s u per - 
ior but differences not 
statistically signifi- 
cant. - 

No significant differ- 
ence between groups. 
Analysis of data shows 
that although students 
trained with 6SMT had 
lower "academic poten- 
tial" TGCT/ARI scores) 
thar\^ntrol groups, 
th%y performed as- well 
as controls. Report 
does not describe equip- 
ment used to train 
control groups. 


OeRau-H and Parker, 
s 1969 \ 

. \ 


„- 20 
\ 20 


54.4 85.5 
54.9 • 83.2 

o 

54.6-; ' " 85.8 


* 








• 




* 


» • 

* 
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A-i/A r ; 



SIMULATOR 



COURSE 



COURSE 
LENGTH 
(STANDARD) 



EQUIPMENT 



TYfEOF 
MSiRUCnM 



ec-ii 



APQ-126 radar for AD 7 
aircraft 

Air National Guard 
Buckley Field, CO 



Mohawk Propeller System 
(53 C5l)» OV-1 Airframe, 
Repair "Course ; 
U.S. Army Transporta- 
tion School , Fort 
Eustis, VA . . 



Hydraulic and flight 
control system, T-2C 
aircraft 



CJT 



3 hrs 



32 hrs 



Engine, power plants 
and fuel system 



24 hrs 



Environment/utility 
system 



32 hrs 



AET: 



SIM: 
AET: 



SiM: 



AET: 



SIM; 



AET: 



^On-the-job training; length not specified 
( 2 'll qualified, 6 untrained. 
' 3 *40 items, multiple choice" 
(4), 



p<*rt id 



Mock-ups and • x 
breadboards; 
conventional 
classroom 
instruction 

EC-II ' 

Arresting gear and 
speed brake 
trainer, main and 
. auxiliary landing * 
gear trainer, - 
flight control 
trainer. 
NAS Chase Field 

Elevator and ele- 
vator trim panel , 
aileron and trim panel • 
hydraul ic speed brake 
panel, landing gear 
panel rudder and 
rudder trim panel • 
wheel brakes and 
flaps panel. 
NAS Meridian 

* Fuel systems- train- 
er; engine ' % ■ 
NAS Chase Field 

Fuel panel; 

DC electrical start 7 

and run panel 

NAS Meridian ' / 

• / 

Heat and ven^/' 
training unit; 
seat, / 
NAS Chase/Field 



AET ; 
Simulator 

4 

AET 



Simulator 

AET > 



Simulator 



A£r 

/ • 
Simulator 



AET J 



pral performance test on T-2C\aircraft at end of course; aircraft p/rt Identification, 
knowledge of naintenance nanuak situational troubleshooting. Scored J>y examiner from 
Naval Weapons Engineering Suppoi\Act1vity. 



tumwMM 



28 



33 
17 



13 



19 



13* 



16 




SCORES 



87.9 



THE 



93.7 

End of (3) 34.8 
course " 

Performance^ 
- ' 82.6 



r(3) 



35.5 



End of 
course - 

Performance^ 
76.2 



End off3) 3^ j, 
course 

Performance^ 
69.7 

End of* 3 * 31.9 
course 

Performance^ 
74.1 



ATTITUOES TOWARDS 



AIl~3udgeone*rning to 
be'.easy; 90 percent 
reconmend simulator for 
training 



Students favorable; 
Instructors neutral to 
negative 



Students favorable; 
.Instructors neutral to 
negative 
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COMMENTS 



Performance measured on 
same practical exercises; 
difference in favor of 

^Imulatorjtatlstlcally 

significant fp»07000f) 



REFERENCES 



Spa ngenbergl 9747 



Dorst 1974 " 



♦Difference In favor of 
simulator statisti- 
cally significant 
(P'0.10) 



*D1ffe$nce 1n favor of 
AET statistically 
significant (p«0.1C) 



Wright and Campbell, 
1975 



Wright and Campbell, 
1975' 



Wright and Campbell , 
1975 



A-3/A-4 



SIMULATOR 



it\l l (continued) 



EC-UVLP 



0) 



(2) 
(3) 



COURSE 



Weapon control system 
mechanic. Block V], 
radar 

towry AFB, CO 



Motor Transport School 
Marine Corps Base 
Camp te,jeune, KC 



Pilot familiarization 
for T-2C aircraft 
NAS Pensacola, FL 



Nayal Flight Officer 
familiarization for 
TA-4C aircraft, 
NAS Pensacola, a 



COURSE^ 
LENGTH 
(STMNMRO) 



60 hrs 



18 hrs 



(2) 



11 hrs 



(2) 



EQUIPMENT 



SIM: Environmental 

simulation panel; 
environmental and 
utll ity panel; * 
sear, 

NAS Meridian 



AET: AN/APQ 126 

radar for A-70 
aircraft 



SIM: EC- J J 



AET^rfo'bile training 
unit, ' 
chalkboard 

SIM: EC-WLP 



(as above) 



Large panel version, interfded for class demonstrations. 

Eight (8) lesson units, e.g., electrical systems, Instruments, ejection system. 
Naval Flight Officers. 



r: 



0 ^ 



COMPARISON 



r tlHKNTS 



SCORES 



TIME 



ATTITUOES TOWAMS" 



HEf€H€MCES 



Performance 

tndV (3) 
;course 

' m 
Performance * * 



84.5 
35.4 




30 NFO'S 



ERIC 



opera- >. 
checkout 
(95 Items); 
solving 8 mal- 
function problems 

Normal opera- 
tions checkout 
(N.S); solving 
malfunction 

grcu!e=s {,*;.$) 



fits Favorable; • 
instructors neutral, to 
negative 



'Students neutral to 
favoraMe; 2 Instructors 
favor simulator; 3 cau- 
tious ' 



Students and Instructors 
moderately to highly 
favorable to SIM (£C\U) 

-Instructors favor simu- 
lator over other train- 
ing aids 



HP 



(as a*bove) 
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No significant 
difference between 
AET and simulator 
group 



Cost estimate for 
equipment U sets) in 
complete course: 
ShCG3,C00 AET vs 
$169,000 Tor simulators 



No data in paper, EC- 1 1 
judged effective for 
training and recom- 
mended for adoption. 
Project savings of 
$386,000 over 15 years 

Finding based on fac- 
tor analysis of atti- 
tudes 

Author judges training 
with simulator to be 
equally effective to 
use of AET at "school; 
performance on-the-job % 
unknown 



Us Above) 



KcGuirk, Pieper and 
Miller, 1975 
(Also Miller and 
Rockway,-1975) 



Piatt 1976, 



Biersner 1975 
Rlersner 1976 



Biersner 1976 



A-5/A-C 



COURSE' 



COUME 



LEMTH 



EOUTMEMT 



Automated Electronics 
Maintenance Trainer 
(AEMT} 







AET 

SIM: Conventional FM 
tuner 

AET 

SIM; • Primary power 

. control for AlM-64 
manual test equip-* 
* ment for AN/AtQ 
100 airborne EW - 
transceiver 



AET 

SIM: AUM06B senU 

automatic test set 
for ALQ-126 EW 
transceiver 

AET 

SIM: Visual target acqui- 
sition system (VTAS). 
helmet -noun ted sight 



^^^^^^^^ " % " * v * * ' * 






> 






COMPARISON 


< 






RnSSrs 






ATTITUOEI TOWARDS j v 


COMMENTS 


REFERENCES 


p ; 


• v v .- 

\ 

- *: . ^ - v 






Provides CAI and CHI 
servicer to multiple 
student stations; one 
instructor station 

No evaluation reported 


Hod rick, Kanarick, a 
Daniel and Gardner, * A 
1975 J 

" * *T 




4 


^ X 


<• 


Evaluated favorably by » 
3 Navy instructors for « 
fidelity of simulation 
and usefulness for 
training 


Modrick, Kanarick, * *- 7 i 
Daniel , and Gardner, • * 

1975 


1 *- - * • 
4" ' % 

If* *" *- 

F, v 




• * 

* 


I 


As above* plus com- 
ments by attendees at 
demonstrations that 
M A£MT approach ap- 
peared to provide a? 
good if not better 
training effective- 
ness than is achieved & 
Using operational hard- 
ware," (p-27). No per- 
formance data 


Daniels; Datta, ' * " 
Gardner, and Kodrick, 
1975 

F " - -- 


*£. ' 








No evaluation reported 


Kodrick, Kanarick, j 
Daniel, and Gardner?, ^ 
1975- i 

* r * ' ' 

Hodrick-, Kanarick, • 
Daniel, and Gardner,- 
1975 

«*■ ' * 
*.' ' 

i 




V 

f: ' 

f ' J, 
t 

v f - - 


< 

* V 


"V" — — ; - 


*« - 

I 


No evaluation reported 


f ^ ^ ^ " 






I 


7 






i 














> 






1 • , 

« % : > 
% » /- i 

) 


1 " ' • 

£ ' o , 








y 
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X-7/A-8 



SMUUTOft 



COURSE 



NURSE 

■ standard 



EQUPMEUT 



TYfEIF 



Generalized Maintenance 
Training Systen (GMTS)- 
Rigney Systen 



"ERIC 
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AN/SRC-20 UHF Voice 
Comunications Systec 
ij Advanced Electronics 
i 'Schools, Departnent 
^facility, JUval Schools 
Cot*and, San Diego, CA 

WSPA-66 radar 
repeater, '* 
Naval Mobile Technical 
Unit 5, San Diego, CA 



AN/tfSC-3 transceiver 
for.fleet satellite - 
^cocouni cation sysren. 
Advanced Electronic 
School Division 
Service School Coracand, 
San Diego* CA * 



4 days 
(16 hrs) 



*AET ^ 
Simulator 



AET 

Simulator 



COUP ARSON 



mm 



20 



10 




Average solution 
tine per problem 
about half that - 

above 



ATTITIIIES TOW AMI 



Students favorable or 
."very favorable 



Students favorable or 
very favorable 



1 la 



Average solution tises 
per problen (N»l 1 ) re- 
ported for students 
trained with sitaulator; 
no baseline -data for 
comparison 



loproved. low cost version 
using a UCSD Pascal , 
.high-level* transportable 
coeputer language* 
Training effectiveness to 
be determined 



DEFERENCES 



Rigney, Tbwne, King and ^ 
Woran (Oct 1978) 



Rigney, Towne.- flora n 
Mishler (1980) ' " 



Totifie and ftinro {1981) 



LBIITH 



fault Identification 
Simulator (FIS) 



Hagen Automatic Boiler 
Control t 

Fleet Training Center 
San Diego, CA . 



5 wks 



AET ^Pneuwtic 

Maintenance Si*u- 
lator"Jand Boiler 
Control replica - 



SW: 



FIS and Boiler 
Control replica 




Actual eiiufprent tracer • 

Actual components activated by pneumatic and electrical signals 

Individualized* self-paced Instruction compared to conventional instruction above * 



STUOEMTS 



SCORES 



ATT1TUOES TOWARDS 
SNUUTKS 



REFERENCES 



5 weeks 



* U' 



"sane as above" 



2.4 weeks 



(3) 



St*«y in Kinlude (1979) J 

'"1 



9 

-ERLO-r 



Ml- 



SWUUTM 



COURSE 



COURSE 
LEMTN- 
(STMRMRO) 



EQUVMENT 



6883 Converter/Flight 
Control Systems Test 
Station (for F-Ul dir. 
craft) ■ 



V 



Mil Avionics 
Maintenance 



(1) 



Tests 

1. End of course^ , . 

2. Projected job proficiency, Part I 

3. Projected job proficiency, Part II 

4. Ratings by supervisors on job * 
performance after 2-32 weekfl^on job. 

1 I 



»1. 



6 days 
(special 
block in 23 
weeks course 
for this 
test) 



: AET: 



6583 Converter/ 
Flight Control 
Systems Jest 
Station 



SIM: 3-dimensional siou 
. lat'ion of above 



COMPARISON 



NO. OF 
STUOENTS 




56 




SCORES 



Assistance 
needed by 
Tests 1 ' instructors 



1. 23.1 

2. 13.9 

3. .23.5 

4. Sane 
• 

t. 22.8 

2/ 14.0" 

3. 23.0 
4:. Same 



2.3 



2.3 



TIME 



54.2 nin 



54.8 nin 



ATTTTUOES TOWARDS 
SIMULATORS - 



StudejJts^ffvofabJ e ; 
instructors neutral to 
slightly favorable 



This Test Station used 
only two days, in regu- 
lar 23 weeks course 
{6 days in this test). 
Test shows equal effec- 
. tiveness at school and 
in follow-up on" job 2 r 32 
weeks later, based on 
supervisors' comments. 
Study also says simula- 
tor costs about one- , 
third that of actual 
equipment to acquire 
and use. 



REFERENCES 



Cfccinelli, Harmon, 
Kel.ler and KottenstetteL 
(1980) 



Ida 



•A-13/A-I 



SIMULATOR 



COURSE 



COURSE 
LENGTH 
(STAHOARO) 



EQUIPMENT 



Paper and pencil 
training aids 



(5) 



Trainer- Tester 



Purtchboard Tutor ' 



Pap'er and- pencil , 
training aids 

Trainer- Tester 



Custom-built 
simulator* 0 ' 



Flow Diagram Trainer and 
Automated Microfiche 
Terminal 



Basic .electronics 
U.S. Naval Training 
Center, Great Lakes, 
!L 



10 wks 



Radar repair 



Army Signal School , 
Fort Monmouth, NJ 



JDA radar display 
HMS Collingwood 



9 wks 



2 days 



AET; (1) . 

AET * Trainer-Tester 

AET ♦ Punchboard Tutor 

Taped lectures(N=4) 
Yrainer-Tester 

Custom-built simulator 



Panel board simulator 

Microfiche projector 

Equipment mock-jp 
(JDA radar simulator) 



(1) 

(2) 

13) 
(4) 
(5) 



Push-pull, three-stage transmitter superheterodyne receiver^ twelve 45-minute classes for each 
50 multiple-choice items 
Grade assigned by instructor 

15 multiple-choice items m vv 

Simulator malfunctions, tests and measurements on specially prepared paper layouts of equipment 
developed by Van Valkenburgh, Nooger, and Neville, Inc., 1954 i 

^locally designed to be* more realistic than Trainer-Tester, uses schematic drawings 





COMPARISON 








Z NO. OF . 
2 ..STUDENTS 


SCORES 

- ^ 


TIME 


ATTITUDES TOWARDS 
SIMULATORS 


. COMMENTS 


■ REFERENCES 


230 


Exam>,{ 74 
Lab 80 

Trouble^ /flV - 
shooting* .'63; 




Students believe aids 
Improved troubleshooting 
-skills. 


No significant differ- 
ence between groups 
trained differently 
when* tested later in 
Advanced training 


Cantor and Brown, 1951 


f \ 230 • 


Exam 75 
Lab tfl 
TrouDle- 
-shooting ' 64 




Instructors prefer Jab 
work to Trainer Tester; ' 
least accept Punchboard 
Tester * 


Conwunications (N s 126); . 
traiper-Tester group 
(N=210) superior on x 
laboratory grades 




; 230 • 


Exam 76 
Lab 81 
Trouble- 
shooting 6G 






< 




6 groups 


* 




Effects treasured at the 
end of the course by 
performance test (find 
malfunction in'actual 
radar components) and 
written tests 


Glals, 1967 \ • 

Si* « 


N«26, 27 each 








Main finding is- that 
lectures on troubleshooting 
improve effectiveness 
of pa^er simulators. 
Custom-built simulators 




12 
\ " 


> *■ • 

No. of checks 
to find faults 
fault before after 


Average tirre to 
find faults^ (min) 
Fault before after 


_ . 

- 


are more effective than 
Trainer-Tes'ter \ 

Training with panel Jboard 
tra'lner and microfiche 
'projector; before and 


Cunningham, 1977 




1 10 - 4 

2 16 . 5 

3 ~ 5 


1 , 18* - 5 
2, 21 6* " 
3 6 




*fter tests with JOA 
radar simulator; no 
comparison with AET^ 






r. 
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;A-15 



SIMULATOR 



Computer s 
training i 
(computer- 
tion) 



inflations for 
n fault diagnosis 
assisted instruc- 



COURSE 



Aircraft power planf condi 
tioning and testing 

Institute of Aviatiorf- 
University of Illinois 



0) 



Fault-finding performances on two 
aircraft engines on test stand*. 



COURSE 
LENGTH 
(STANDARD) 



3 special 
training 
sessions 
(total of 
six hours) 
in semester 
course 



EQUIPMENT 



PEOF 




Context free simulation 

,ContexU«t>ecific simulatiort 

Instructional TV film on* 
troubleshooting engines 

Continental 0-300 recipro- * 
eating (on test stand) * 0 



Lycoming 0-235 reciprocating 
engine (on test stand) 



INST 



Instruction- 
al TV 

Context- 
fault diag- 
nosis 

Context- 
specific . 
fault diag- 
nosis • 
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COMPARISON 



NO. OF 
STUDENTS 



SCORES 



(1) 



Perform- Percent of Percent of 
ance appropri- inappro- 

•score — =*te prlate-— 

actions actions 
4.4 , 44« 0.9% 



4.0 



4.0- 



28 



30 



0.7 



1.4 



TIME 



(1) 



Time/ 
Problem 



1.3 hrs 
1.8 



2.2 



ATTITUDES TOWARDS 
SIMULATORS 



Attitudes (5 most favorable) 

.4.2- 
4.4 

3.8 * 



COMMENTS 



Transfer measured from 
method of training to 
fault finding, 5 mal- 
functions* in 2 actual , 
engines on test stand. 
Training with instruc- 
tional TV yielded best 
troubleshoo ting s perform- 
ance* judged due to 
similarity of training 
of training with test. 
Evidence that computer 
simulation transfer also 
found but less pronounced. 



REFERENCES 



Johnson* 1980 



i 



ERIC 
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APPENDIX B 
PROGRAM COSfS OF NON-STANDARD SIMULATORS 



_ . APPENDIX B 
PROGRAM COSTS OF NON-STANDARD SIMULATORS 
A. SUMMARY " 

Table B-l provides^ summary of costs for nine non-standard 
simulator programs for which data were available. Table B-2 
shows the -normalized- costs of these programs. Normalization 
involved two adjustments bo the program costs. 

(1) In order, to provide r consistent base for comparison 
among programs, recurring production costs yere di- 
vided by the quantity fabricated. Thus; each program 
y 111 5 eflect the costs that (hy pothetically) would 
have- been incurred if only one trainer had been pro-7 
duced. 

(2) Two of the research programs (the 6883 and MA-3" Test 
Benches) incurred significant costs for evaluations. 
These costs were excluded so that each program will 
reflect the costs that (hypothetically) would have 
been incurred if they had been intended for 
main-line training. 

The values shown in Table B-2 provide the basis of the percent 
ranges of the different cost elements shown in Table 15. 

The remainder of this Appendix provides a short discussion 
of each of the nine programs and a more detailed display of 
their costs. , - <t 
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r -» , " TABLE B-l, NON-STANDARD SIMULATORS, SUMMARY OF TOTAL PROGRAM COSTS . 
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K TABLE B.-2. NON-STANDARD SIMULATORS^ SUMMARY OF NORMALIZED TOTAL PROGRAM COSTS*'-'-* 
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B.' RESEARCH PROGRAMS 



/. 



I The three research programs (simulating the Navy A-7E HUD 
and MA- 3 and the Air Force 6883 Test Benches) have had mixed 



/histories. Each was conceived as a vehicle- for an extensive 
I evaluation, comparing the effectiveness of maintenance sirau- 
j latlpn as an alternative to actual' equipment for training. 



/ 



and there is no evidence that they were intended for use in 
regular train ing^programs* However, current planning envisions 
using, the MA- 3 Test Bench for training* These three devices 
.are' the only ones, to date, that address simulation for inter- 
mediate (as distinct from organizational) echelon maintenance 
training* * . 

Each of the trainers is a unique 3-d,imensional device, 
and each has had a different principal contractor* For the 
two Navy devices., the contractors had no prior' experience in 
developing- similar systems* The contractor for the 6883^ (Honey- 
well) previously had built one model of a 3-dimensional mainte- 
nance simulator*' 



1. Honeywell 6883 Test Bench 
- « The 6883 device wa^ procured as part of a continuing main* 
. tenance simulation research project of the Technical Training 

Branch of- the Air Force Human Resources Laboratory (AFHRL) • 
\ It .is the only one of the three research devices that has been 
extensively evaluated; results have been published recently in 
Deignan and Cicchinelli (2980) and Cicchinelli, Harmon, Keller, 
and Kottenstette (1980)* This evaluation addressed the simulator 
' for training operations and maintenance. of the 6883 Test Bench*, 
A .standardized 2-D system, also simulating the. 6883, has recently 
been 'delivered and will be evaluated for training operation and 
maintenance of the test bench* 

Program cost information (Table B-3), based on contract 
actuals and in-house costs, was obtained from the AFHRL program 



TABLE B-3. 6883 TEST BENCH (Dollars, Thousands) 
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office. Honeywell contracted for this. program at roughly the 
same"time as its VTAS system was delivered to -the Navy. There 
appears to be little similarity between the ttto, fend it is 
unlikely" that the- prior VTAS experience proved significant in 
the 6883 program. • » 



I. A/7 



E HUD Test Bench 



' . The kfJE AMD (Head-Up-Di splay) Test Bench as one part of 
a Navy .research program that envisioned development and evalua- 
tion of -^six 3-dimensional maintenance, simulators — one eacn . 
for training in three types of skills (electronic, electro- 
mechanical, and mechanical) at two maintenance echelons (orgarii 
zational and intermediate)'. In addition to evaluation of these 
dif ferent^applicat ions of simulators,, the research program was 
*to resuft "in developing procedures for formulating sinfalator 
develppment specifications. * 

The A-7E HUD is rt the intermediate echelon, electronic 
skills portion of this research program. To date, the device 
has not been evaluated. Aft evaluation program was designed 
and the simulator was delivered to the .training siteJt However, 
the program was not initiated, and its future is in "dfeubt. 

This was the first device to be initiated under \th^ 
research program, an$ the first of its kind-fcr the Nal/y. 
Personnel involved with management of the program (the mw»an . 
Factors Laboratory of the Nava'l Training Equipment Center) are 
frank to state that it was a learning experience for them in 
terms of simulator specification (one objective of the research 
program) and development procedures. The device was initially 
litaited to simulation of the "day" , version o,f the HUD that 
was a mature A-7E system at the time. However, a FLIR (forwchfd 
locking-infrared) version was in development at that time, and 
the simulator . program was later expanded to incorporate the 
FLIR capability. This^ opened the program to two problems -- 
modification and concurrency. 



. Program costs (shown in Table B-4) were developed by the 
Human Factors Laboratory from contract records and the expertise 
of La! oratory personnel. The costs' show the impact of the 
decision to simulate the FLJR capability.. Six contractors were 
involved in the program, three of these (including the A-7E 

-prime contractor), because of, the FLIR decision, over one-third 
of the program total costs (and over 40 percent of contracted 
costs) resulted from the PLIR modification. Unfortunately, .. 
there is no way to separate FLIR-associated costs into those" - 
ar-ising from modification, per se, and those arising from con- 
current development of the simulation and the. operational - 
equipment/ Program office personnel believe the latter was a 
significant element', and state that during several intervals 

•development of the simulator was ahoad of development and docu- 
mentation of the operational 'equipment, 
if 

3 * MA-3 Test Benc h 

'The MA- 3 is a seopd element of the Navy, maintenance simu- 
lator research program, in this case addressing training -f 
intermediate echelon, electromechanical ' skil Is. This proqram 
was initiated more than two years after the A-7E HUD and rcn<- 
fited. siqnificanWy from the earlier experience, accordinq to 
program office personnel. The device was delivered durinu the 
summer of 1980 and is currently undergoing evaluation that 
should be complete in June 1981.. 

Proqr.im costs are .shown in Table '*-*. These .data were 
developed from contract records and expertise of personnel 
fromthf Human Factors Laboratory that managed theproqram. 

. ' The simulator *ias one noteworthy characteristic, differ- ' 
r;nt from any of the other non-standard devices, that is com- 
pletely uncelatr-ri to its rolo as a research vehicle. it is 
the only device of its class to pr.Wde training that' is eot- 
specific to a r.mqle weapon or support system. The MA- 3 is a 
universal stand used throughout the Navy for on-shore testing 



TABLE B-4. A-7E HUD TEST BENCH (OollarsY. ThoUsand^^^^lv 
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TABLE B-5. . MA-3 TEST BENCH ( Dol 1 ars ,'" Thousands ) 



Contractor 



Task/Function, 



Application, Inc. 



In-House 
Engineering 
and 
Support -~~ 



Initial 
Contract 



Program 
Expansion/ 
Modifi- 
cation 



Cost 
Overrun 
and 

Spares^ 



Seville 
Training 
Cost/ . 
Effective- 
ness 
Evaluation 



Non-Recurring Cos t 

Front End Analysis 
Task Analysis 
Other 



Design and Development 
Hardware 

Software/Courseware 
Technical Data 
Other . > 



Hardware Fabrication 
(non-recurring) 

Test and Evaluation 

J Program Management 

Total Non-Recurring 



Recurring Cost * 

Production 
Hantojre Fabrication 

(recurring) 4 
SpeciaT Tools/Test 
^ Equipment , 
Initial Spares 
Other * 



Logistic Support 
Interim Maintenance 

Support 
Other 



Initial Training 
* Total Recurring 



Program, Total Cost 



40 



40 



40 
80 



80 J IW 



80 
38 
42 



20 
7 

13 



111 

29 
82 



11 

m 



42 
42 



14 

132 



47 
10 
37 



? 

56 



90 



90 > 



14 



14 

" 7 
63 



28 

28 ' 



29 
12 



J 25 



28 



54 



H 



160 



* Including insulation and checkout. M 



no 



9C 



B-ll 



of all generators and constant-speed-drives that comprise air- 
craft primary electric power systejns. (A related test stand, 
"~Eh^~OTA--2r—servos- tke^same f unct ion for all primary aircraft ' 




power systems on board ship.) As a rep alt , knbwleagB of hoth' 
* its operation and maintenance is a widespread requirement and, 
if explcpited,^ a variant of this simulator (modified for main- 
line training) might satis.fy an extensive training requirement. 

The MA-3 consists of two distinct components the t$st 
stand itself and associated equipment (such as electronic test 
sets and oil coolers) to adapt its use to the range of different 
generators and constant speed -drives that ar£e used on various 
J Navy aircraft. The simulator 'was designed with this in mind 
and maintains a separation of these two components.. Although 
the current simulator was built^ specifically for training .AV-8A 
maintenance personnel, the contractor has provided the following 
rough estimates. 

• Modification of, the current simulator system to allow 
simulation of other generator/cpnstant-speed-dr ive 
combinations would cost $15,000 to $20,000 (including 
computer programming and fabrication) for each combina- 
tion. 

m Follow-on units in lots of two of the current configura- 
tion (including hardware , 1 software , and documentation) 
would cost $170,000 p.er'unit. 
V Two points should be noted with regard to the second con- 
tractor ^est imate. The first_is. that the current simulator is 
configured for reseaVch and it can be anticipated that quantity 
production of a device that did not contain features needed for 
research would c6st significantly less. The'.second point i$ 
that there is an apparent inconsistency between the contractor's 
estimate -and the costs dev.-l^ped*by Hum^n ractors Laboratory 
personnel. The iat't'er costs attribute only $99 thousand to re- 
curring device .product ion and $-145 thousand to recurring program 
costs (and include .ill non-cccurr inq fabrication costs that,. in 
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in fact, were incurred). We have no explanation for the differ- 
ence in estimates. However, the lower Laboratory estimate is 
consistent with data obtained on "other programs. Either esti- 
mate total:?! to much* less than the cost incurred in the research, 
program, and the bulk of this cost reduction can be attributed 
to the-high proportion of non-recurring costs associated with 
one-of-a-kind production. 

* 

C» A- 6 TRAM DBS > . 



The A-6 TRAM DRS (Defection and Ranging System - AN/AAS-33A) 
simulator was- initiated as a research program, i.e,., as one ele- 
ment of the Navy research program that includes the A-7 HUD and 
the MAr-3 Test Stands. However, this aspect of the program was 
modified at an early stage, and the simulator was adopted as' 
one of the primary devices for organizational echelon mainte- 
nance training of the electronic portion of the DRS. Two units 
were built and shipped to the two A-6 training detachments 
where ttiey were placed in main-line training after a relatively 
short checkout and acceptance test period. ' * " 

This appears to be one of the more successful non-standard 
simulator programs. The deyices cannot be considered, high cost 
and have been well received by t-.raininq personnel. However, 
certain features or the' device and its procurement point up 
problems that seem to recur with maintenance training simulators. 
Four ft£ . th9f e are discussed below. 

• The simulator, provides only for organizational mainte- 
nance training on the electronic portion of the DRS. 
^ As a result, it did not relieve^ requirement 'for using * 
-operational equipment for training on the mechanical 
portion of the DRS. While the Existence of the simula- 
. _ t0r reduces the training 'load placed on the operational 
m equipment, it does not necessarily reduce the amount'- 
or cost of actual equipment required for training. 
< . ' ' ' 
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This model of DRS is peculiar to Vhe A76 aircraft. A-6 

training is conducted ^t*%only N two 'locations , and the 

two devices serve to fully' satisfy the training require- 

* \ * ^ 

ment. This is typical of aircraft systems 'today, even 

those that are procured in large numbeYs. When it Is 
considered " that the ma jor"portion of simulator program 
cost is nonrecurring, in nature (approximately 75 peV-* 
cent in this case) there appears to be little promise 
in reducing thei^r costs/ except by use of standardized • 
^systems where high development costs (such as for the * 
software systems)' can be spread to a number o^ training 
applications. ' \ 

.This is the first device of this type to be contracted 
for by Grumman", and its capabilities and design }includ- 
ing the software packaqe) are highly tailored to this 
single training application. Configuration cfianges^and 
modifications are common to current combat aircraft &nd 
even apparently minor ones may result in major change^ 
to maintenance simulators whose, costs m^y be a major v 
portion of those required for developing a new device* \ 
Decisions regarding the timinq of training device deliv-y 
eries are critical* in providing training on new. or modi- 
fied , operational equipment,. .Whatever types of training 
devices are used, they foust be in place before that 
training can commence, and training must be provided to - 
personnel before the operational equipment can becbme 
an effective part of the Jiorce. Since simulation > 
trainers require t-heir own development, this must occur 
concurrently with development of the operational equip- 
ment.. However, the operational .equipment is subject to 
frequent modif icat'ion during development and for a con- 
siderable period after its fielding. As discussed above, 
these modifications can have a drastic impact on. Simula- 
tor oosts. An example of this impact can be seen in 
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the A-6 DRS program. The DRS tactical configuration 
and the simulator were developed at the same time, both 
by Grumman, and the DRS configuration was not 'finalized 
J at the time thfe simulator program^ was initiated. TKe 
^simulator contract provided an t allowance , for changes 
in the Simulator that 'were the result of anticipated 
% ' ^modifications to the operational equipment, and at the 

time of final delivery the cost of tKesc .ihang'es amounted 
to 35 percent of the final Grumman cohtra^t value. 
Program cost information (Table ^B-6) was obtained from the 
program procurement manager at the Naval Training Equipment, Cen- 
ter. At* present, both trainers have been delivered and placed 
in service; the costr, shown" in Table B-6 appear to be th^-total 
costs" to the government, except for updating the de^r^es to re- 
flect tecent modifications to the operational^equipment. 



\ 



D. 



VTAS 




The VTAS simulator^a its origin io a change in* configu- 
ration'^ the F-4J^arTrcraft that entailed changes in training 
equipment th^n^employed at 'two Naval Air Stations (NAS) . An* 
assessm^ftt of alternative^ for these changes concluded that 
s-ijBulation would costjbetween 60 and 85 percent of the alter- 
natives that involved modifying or procuring additional opera- 
tional equipment far training. 

The contractor (Honeywell) has produced several non-standard 
maintenance simulators." Howfcver , ^VTAS was the first and appears- 
to be quite different from. Honeywell ' s later- programs. The 
relative distribution of program costs among cost elements is 
quite different from that on other programs (including the 
later Honeywell programs). , It is concerned only with a single 
system that is not complex relative to current avionics systems. 
The Navy procurement program manager described the training 
requirement as relatively simple, and the simulator reflects 
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TABLE B-6. A-6 TRAM DRS (Dollars, Thousands)- 



Contractor 


Applied 
Science * 
Associates 


Grumman Aerospace 


Total 
Program - 


Task/Function 


Task 
Analysis 


Initial 
. Contract 


Engineering 
Change 


Non-Recurring Cost 

Front End Analysis 
• Task Analysis . 
Other 


40 
40 




* 

4 


40 

40 t J 
• * 


'Design and 0*»*iopinent 
Hardware 

Softwarv/Coursewa re- 
Technical Data 

*\ otHer 


\ 


148 * ' 
. 10 
85 
47 
6 


' 163* 
I 163 


31! 

10 
248 

47 
6 


Hardware Fabrication 
(non-recurring) 

lest ino tviiuiiiCfi { 
Program Management 
. Total Non-Recurring 


r 

40 


> 
40 
* 195 


* 

163 / * 


7 

398 


Recurring Cost 

" Production r 

Hardware Fabrication . 

(recurring)* 
Special Tools/Test 
Equipment 
• Initial Spares 
Other . 




92 

35 

13 
44 


* 

*> 


♦ 92 

* 35 „ - 

13 * 
• v 44 


Logistic Support 
Inter in Maintenance 

Support 
Other 




11 
11 


- 

• 


11 • 

. 11 
• 


r 

Initial Training 
' Total Recurring 




17 

120 




*120" 


• Program Total 1o*» 


40 


315 


163 


sift * ". 



Including Installation and checkout. 



the limitation (providing eight malfunctions, that are integrated 
% *into the contractor-proprietary software system)*-. x * 
Cost information (Table B-7) was obtained from tfoe con- 
tractor; In addition to ,the, contract' costs, the procurement 
program manager has estimated that, between ond-hal£ and three-- 
quarters of v a man-year, was expendecl by in-housfe civilian and 
military personnel on various functions (including front-end 
analysis and program management). 

E* AT-TRAINER * 

The AT-Trainer takSs its iiame ftom 'its purpose of provid- 
ing traininq ifi aljt equipments maintained by tfye Natfy aviation 
technician (AT) rating on three series of the F-4 aircraft^- 
the communication/navigation/identif icatiori and the" electronic 
.cbuntermeasures suites.. It was begun as a small scale in-hou$e 
'project at th'9 North Island Naval Air Rework Facility to pro- 
vide ^simulation of % one rnodel^ of^UHF communication equipment on 
the F-4N.- Since that time, it h*as expanded thcough a? series. * 
<>i program .changes to encompass all AT-maintained equipment qt 
tjie F-4N, F-4J| and.RF-4B and the inertial navigation equipment 
of ' the RF-4B ma.intaijied, by the aviation electriaian '(AE). rating^ 

This simulator provides only fjor organizational ^inte- 
■ ^ * ' ** 

nance 'training that 1 is typically confined to troubleshooting 

of installed equipment.* This involves a large number of 

equipments the controls and iindijators of-which are located on 

the aircraft instrument panel: the physical configuration of 

the simulator constitutes an extensive mock-un of the cockpits. 

The particular equipments simulated, c by aircraft series, are 

shown in Table B-8* * - * * 

> At program completion three units will have £>een built* 

Two of the^se (to be used at Beaufort MCAS.and 1 Oceana NAS) wfll 

,be limited to providing AT training fQr theF-4J. The third 

(delivered to El Toro MCAS) will provide simulation for AT 



TABLE B-7. VTAS (Dollars, Thousands) 

- .0 





» Contractor. 


Honeywell 






<> 


. 


Task/ Function / 

,/ 


Total «. 
Program 




Non-Recurrlng Cost 




• 


Front End Analysis 
Task Analysis, 
Other, 






Oeslgn and Oevelopaent 
Hardware 

SoftMa re/Courseware 
Technical Data 
Other 


74 o A 
19 
38 
17 




Hardware Fabrication 
(non-recurring) 






Test and evaluation 


6 




Progrta Management 






• ♦ Total Non*Recurr1ng 


1J3T 




Recurring Cost 




c 

\ 


Production / 
Hardware Fabrication 
(recurring)* / 
\ Special Tools/Jest 
\ Equipment/' 
\nltlal Spates 
Otjier 


163 
156 

7 


/ 

/ 


Loglsifc Support 
Interim Maintenance 

/ SuppoVt 
' Other \ 




Initial Training 


3 




Total Recurrent; 


166 




\ 

Program Total Cost 

\ 


297 



Including Installation and "checkout. 

\ 



\ 

\ 
\ 

\ 
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TABLE B-8. SIMULATED AT-TRAINER EQUIPMENT, BY AIRCRAFT SERIES 

/ 



Equipment Designation (Function) • 


/ 

Aircraft Series 




F-4N. 


RF-4B 


•F-4J/S 




Cofflfltunf cation/Naviaation/ Identifirafinn Pnnim«nt 






/ 




AN/ASQ-19 (Intearated Flertronirs fpnfr^n 


X 








AN/ARC-159 (UHF Communications) 


X 


/ 
i 
/ 

/ 


X 




AN/ASQ-160 (Integrated Flprfrnnir* fpnr r*l \ 




v / 






AN/ARC-105 (UHF Conmanications) 




1 






AN/ARN-118 (TACAN) 




/ 


X 




AN/AJB-3 (Computer) / 


X, 








AN/AJB-7 fCnmniifprl ' 




i 

» 

! 


X 




AN/ASN-39 (INS) 


X 


! 


x 




AN/ASN-59 (Attitude, Heading, and Reference)* 




/ 

X 






AN/ASN-92 (Carrier Alignment JNS) 




x • 






AN/ASW-25A (Data Link) 






X 




AN/&SV-25B (Data Link) 




X 






AN/ASM-23 (GSE) 


X ■ 


X 


X 




ECM Equipment 


1 








AN/ALR-45 (ECM) 


/ 

1 


X 


X 




AN/ALR-50 (ECM) j 


i 

< 


•X 


X 




AN/ALQ-126 (ECM) . 




X 


X 




AN/ALE-29 (Chaff Disoenser) 






X 




AN/ALE-39 (Chaff Dispenser) / 




X 


X 




AN/ALM-70 GSE (AN/ ALE -^9) 
AN/ALM-164 GSE (AN/ALE -3^) 






X 






X 


X 




AN/ASM-456 GSE (AN/ALR-45) 




X 


X 




AN/ALM-140 GSE (AN/ALR-50)\ / 




X 


X 
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training in all three aircraft series in addition to AE training. 

The multiple simulation capability is provided by "quick chaage^ 

_?5i?^ fc £ - r -fcC0ftf igure -the-simqlator to each aircraft. 

The El Toro unit serves as a test bed for the program. It 
was aelivered as a partially completed- device in^mid-1979. 'Fur- 
ther capabilities {in terms of the equipments simulated) are 
delivered as they are developed, and the system will not be ' 
completed 'until mid-1981. The Ckreana apd Beaufort units'were 
delivered in December 1980 vithdut ECH simulations; this capa- ' 
bility will be ^instal led during 1981 as a field modification. 

Throughout its term, the AT-Trainer program has been 
carried oh strictly as an in-house activity. * The tasks involved 
include development of simulation hardware (including some of 
the data processor) and the software operating system. Ac a 
result, it is a unique system. Apparently, there are no plans 
to carry these developments any further or to employ them in 
otn.er simulations. 

, The, AT program costs are shown in Table B-9. The. program 
manager feels these values may understate true costs of. the 
project, because „many of the individuals involved provided 
extensive unpaid time. In addition, the costs are probably 1 
not comparable to those for other simulator programs. The 
accounting methods employed will differ from those used by 
con-tractors so that' various categories of costs may have dif- ^> 
ferent meanings. 

• •* 

F. INTEGRATED RADIO ROOM 

< * 
This is one of two programs to incorporate simulation into 

the initial desiqn of the training program for a new ma*for 

weapon system. A total of four simulators are employed in the 

complete Trident training system. Two are associated with the 

Integrated Radio Room (the communications system) whale the 

other two are associated with pneumatics and are standardized 

simulation systems. • 

r*-20 
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TABLE B-9. AT TRAINER (Dollars, Thousands) 



Contractor 


north Island Naval Air Research 
Facility 


Task/Function 


Initial 
Prog ran 


Kodif- 

cations/ 

Updates 


Total 
Prog ran 


K *»*Recurrinq Cost 






i 


Front End Analysis 
Task Analysis 
Other 


20 
20 




20 . 


Design and Development 
Hardware 

Software/Courseware 
Technical Data 
Other 


920 
355 
415 
ISO 


95 
50 
45 


1.015 
405 
460 . 
150 


Hardware Fabrication 
(non-recurring) 


267 & 


9 


276 


Test and Evaluation 


Q 

O 




8 


Prooraa Management 






40 


Total Non.Recurrlng 


1.255 


IC4 


1 ,359 
. » 


Recurrinq Cost 








rroauccion ♦ 
Hardware Fabrication 

(recurring)* 
Special Tools/Test 

Equipment 
Initial Spares 
Other 


440 
410 

30 


I! 

' } 


451 
421 

30 j 

j 


logistic Support 
Interim Maintenance 

Support 
Other 




1 


1 

! 


!nit*al Training 


30 




30 


Total Recurring 


470 


n ! 


431 * 


Prograa Total Cost 


1.725 


_ 

115 


1 .840 J 

i 



* Including installation and checkout. 



Includes S^CS 'or depot repair facility. 
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Trident communications system personnel are responsible 
for both operation and maintenance of the system; an<3 both 
functions are trained in the same course employing the same 
training equipment. Three major training devices are used, 
designated A, B, and C; two of these devices (A and H) provide 
both operation and maintenance training, while trainer C is 
devoted' to maintenance training* 

• Trainer A is a reconstruction of- the on-rbdaFd radio 
room* It consists of a complete set of the operational 
equipment (along with training-unique equipments) to ^ 
provide operator team and watch-standing training and 
hands-on, on-equipment maintenance training. 

• Trainer B is a reconstruction of a part of the radio 

only simulation equipment. Its function is 
< 

to provide individual and team operations training and 
training in the system fault isolation and diagnostic 
capabilities of the operators' console, and the use of 
the built-in test equipment 

• Trainer C consists of both simulated and tactical 
equipment. It is a series of part-task-trainers to 
provide training in troubleshooting and fault isolation 
procedures at a module/component level and hands-on 
preventive and corrective maintenance. 

This program provides a definitive demonstration of both 
complementarity and substitutabi 1 lty between simulation and 
operational equipment. Two"^rainers are used for operator 
training, and all throe for system maintenance training. Early 
studies of Trident training requirements identified two alterna- 
tives for radio room training. One alternative was the current 
program combining operational equipment and simulators. „ The 
second alternative proposed only operational equipment trainers 
(.consisting of three of the current A-frainers). 

RCA is the contractor for the operational equipment as 
well "as the three trainers? both the trainers and the operational 
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equipment were developed concurrently. The training equipment 
contract is RCA's first experience in developing simulators. 
One_could expect the simulation trainers to be closely patterned 
after the operational equipment and, perhaps,' to be quite dif- 
ferent from other non-standard simulators. 

Our opinion that these simulators differ from others is 
bolstered by comparing the relative cost of software/courseware 
in this program with other programs. Roughly 15 percent of RCA 
costs are attributed to software/courseware, while Che average 
for the other non-standard systems is clo'se to 35 percent. The 
operational equipment employs extensive data processing, and the 
relatively low software/courseware costs for the simulators 
would be consistent with RCA incorporating its detailed knowl- 
edge of the operational equipment "software into the development 
of the simulator software system— rather than developing a 
wholly new system, as appears to be. the .case of other non- 
standard simulators. 

Simulator program costs (shown in able b-10) were obtained 
from the program office for Trident training (at the Naval 
Training -Equipment Onter). They are based on the original 
cost proposal and the program changes that have been negotiated 
to date. 

* 

G. F-16 

The F-^6 maintenance s lation system is the most ambi- 
tious nprt-standard program undertaken to date. Like the Trident 
Reid><5 Room, the use of simulation was incorporated- into the 
initial design of the maintenance training program and the 
-simulators were developed concurrently with the operational 
equipment. Unlike Trident, though, simulation was developed by 
a second contractor (Honeywell) under subcontract to the weapon 
contractor (General Dynamics). 
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TABLE 


B-10, 


TRIDENT 


INTEGRATE 


ID RADIO 


ROOM (Dollars, -Thousands 


Contractor 


RCA 


ECC 


wci i ton 

Systems 
Cnrpnrat.ion 


In -House ' 






Upera tor/Maintenance 
Trainer 


Maintenance Trainer 


Task 
Analysis 


Program 

Office 

Support 


Front End 

and 
Management 
Support 


Total 
Prog cam 


tTaik/ Function 


Basic 
Contract 


Contract 
Changes 


Basic 
Contact 


Contract 
Changes 


Non-Recurring Cost 


• 










i 






Front End Analysis 
Tisl Analysis 
Other- 










400 
400 




270 
270 


670 
670 


Oeslgn and Development 
HarcMare 

Software/Courseware 
Technical Oata 
Other * 


1,856 
1,238 
431 
187 


307 
204 
89 
14 


1,417 

886 
353 
178 


27 
20 
6 
1 




— — 




3,607 
2,348 
879 
380 


Hardware Fabrication 
(non-recurring) 















— . , 




Test and Evaluation. 


12 




12 










?aL_~- 


Program Management , 


c 1 U 


cc 
ob 


145 


9 




248 


140 


817 


Total Non-Recurring 


2.078 


372 


1,574 


36 


400 


248 


410 


5,118 


Recurring Cost 


















Production 
hardware Fabrication 
(recurring)* 
^SpeclaT^Too Is/Test 
Equipment 
Initial Spares 
Other 


1,1*4 

725 

78 
321 


58 
58 


506 

318 

39 
149 


2 
2 






• 


1,690 - 
1,103 

117 

470 ' 


Logistic Support 
Interim Maintenance 

Support \ 
Other 


47 
47 




47 
47 










94 
94 


Initial Training 


114 


4 


68 










186 


Total Recurring 


1,285 


6? 


621 


2 








1,970 


\Program Total Cost 

\ 


3,363 


434 


2,195 


38 


400 


248 


410 


7,088 



Including installation and checkout 
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The simulators encompass eight of the aircraft systems, 
and are to be used by both USAF and NATO country personnel. 
The simulators are configured as both .flat panels and cockpit ♦ 
mock-ups* The quantities procured, by aircraft system simulated, 
are shown in Table B-ll. Typically, the simulation of one air- 
craft system employs two panels or one panel and a cockpit 
mock-up. 



TABLE B-ll. F-16 SIMULATOR QUANTITIES PROCURED, BY AIRCRAFT SYSTEM 



Aircraft System 


Number of Elements 
Per Simulator 


Number of Simulators 
Ordered 


Panels 


Cockpit 
Mock-ups 


USAF 


NATO 


Environmental Control 


2 




3 


2 


Navigation 


2 




3' 


2 


Fire Control 


1 


1 


3 


3 


Flight Control 


2 




.3 


3 


Hydraulic 


2 




3 


3 


Electrical 


2. 




• 3 


2 


Weapon Control 


2 




3 


3 


Engine Start 


1 




3 

j 


3 


Engine Diagnosis 


2 




3 ' j 


3 


Engine Operation 




1 


I 

3 ' 2 

r 



USAF, in buyinq three sets of the simulator system, will 
recoive 48 panels (of 16 different types), six cockpit mock-ups, 
and 30 processors; NATO countries, in total, are procuring two 
or throe trainers for each aircraft system. 
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# The P-16 maintenance simulator family resembles a stand- 
ardized system* V A common model of processor and a common soft- 
ware system is used for all training applications (panels/mock- ■ 
ups). The F-16 jsystem, along with the AWACS simulators dis- 
cussed below, may well be the first members of a new standard- 
ized simulator" system* One result of these simulator programs 
is Honeywell's development of what it has termed the" "Data Base 
Generator." This is a programming language intended to alloV 
for a simple manual translation^ technical manual information 
into code and the machine translation of that code into FORTRAN ♦ 

The F-16 program provides ^ dramatic example of the prob- 
lems that may arise when operational equipment and training 
simulators are developed concurrently. The initial contract, 
in September 1977, provided for a target price of $7.5 million 
for* development and procurement. Delivery to the Air Force of 
the first set o' 18 panels and mock-ups and 10 processors was 
stipulated for September 1978. 

The simulator design freeze was set for January 1978, and 
the configuration was to be based on thfcv production version of 
the aircraft. However, the technical documentation of the pro- 
duction aircraft configuration was not available at that date; 
the aircraft continued to undergo engineering changes, and a 
lag developed in documentation of the changes. The aircraft 
configuration changes imposed configuration changes and rework 
of the simulation models (panel elements, software, and course- 
ware); the lag in documentation increased the amount of rework 
required to accommodate the changes. 

A recent contract amendment res v et the target price of the 
training equipment at $28.9 million. It is impossible to at- 
tribute a specific portion of the increase to the problems 
associated with concurrency. However, it appears that a siz- 
able portion did arise from this cause. 
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— "Documentation of current program costs includes separate * 

estimates of the costs (in terms of the target) for the train- 
ing devices associated with each aircraft system simulated for 
.each of the seven sets of simulators currently on-contract.' 

. These. estimates are shown in Table b-12. Note that the fcotal 
of non-recurring costs are attributed to simulator set number- 
seven and that the values shown exhaust the contract total 
target cost. (That is, all costs have been a 1 located the 
. individual simulations even though some fraction^ them, 
especially %of the non-recurring costs, are Support funetions 
that* are truly common to all elements of ^thc program.) 
\ ^ The estimates contained in Table B-12 allowed a separation 
of recurring and non-recurring costs for the simulations asso- 
ciated with each aircraft system shown in Table B-13. (Note 
that the method used for separation provided a slightly dif- 
ferent estimate of total program cost.) The resulting ratios 
between recurring and non-recurring costs are quite consistent 
with other non-standard .simulator programs (see Figure 4). 

H. AWACS NAVIGATION 

The Air Force plans to provide training simulators for 
three AWACS sy$teas--navitj«tion, radar, and possibly the data * 
display/processor systems*- The navigation system has been - 
delivered and is in operation, while the radar system trainer 
has recently been placed under contract. Both these simulators 
are contracted to Honeywell. Both are flat panel devices and, 
considering the continuity they afford Honeywell, it would 
appear that they will be quite similar. However, costs of the 
devices will be significantly different; one explanation for 
the large difference in cost is that the naviation trainer 
has a Single student station while the radar trainer will 
have 10. * 
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TABLE B-12. 



F-16 



SIMULATOR COSTS BY AIRCRAFT SYSTEM AND SIMULATOR SET 
(Thousands of Do 1 1 ar s ) 



Aircraft System Simulated 






Simulator 


Set 


* 


- 


Total 


1 

USAF 


2 

Belgium 


3 

Netherlands 


USAF 


Norway 


7* 
USAF 


Fire Control * 


424 


186 


169 


137 


101 


1750 


2767 


Flight Control 


528 


361 


365 


356 


328 


2495 


4433 


Navigation 


321 




240 


229 


. 234 


1923 


2947 


L i ci tr l tu I 


J J/ 


cot 








1 Jn 




Environmental Control 


215 " 




172 


144 


133 


1224 


1868- 


Hydraul ic 


202 


16* 


170 


160 


163 


1004 


1863 


'Weapons Control 


297 


253 


216 


166 


1 21 


"1769 " 


2764" 


Engine Start 


280 


173 


170 


191 


■ 166 


1307 


2287 


Engine Diagnostic 


543 


- 331 


327 


348 


326 


2628. 


4503 


Engine Opera ting. 


401 




257 


273 


249 


1474 


2654 


Total 


3548 


" 1732 


2087 


2265 


2062 


17,028 


28 l 722 b 



. Note; Simulator set number 6, intended for Denmark, was cancelled. Totals may not add 
due to rounding. 

including non-recurring costs. 

^Does not include $165 thousand for proposal preparation. 



TABLE B-13. F-16 SIMULATOR, RECURRING AND NON-RECURRING COSTS 

(Thousands of Ool lars) ■ \ 





A 






\ 


Cumula ti ve 




Aircraft System Simulated 


Total 
Cost 

— — 


Non-" 
Recurring 

LOSt 


R o c 1 1 1* y* i nn 

Cost 


Average 
Recurring 
Cost 


: — 

Fire Control 


2756 


1640 1 


„ 

1116 


186 




Fl ight Control 


4413 


2174 • 


2239 


373 




Navigation 


2956 


1699 


1257 


251 




Electrical 


2621 


1269 


1352 


?70 




Environmental Control 

< * 


1872 


1110 


762 


. 152 




Hydraulic 


1849 


852 


997 


166 




Weapons Control 


2781 


1560 


1221 


203 




Engine Start 


2440 


1125 


1315 


, 219 




Engine Diagnostic 


4635 


2307 


2328 


388 




Engine Operating 


2713 


1227 


1486 


297 | 




Total 


20,036 


14,963 


14,073 

J 


' ! 

i 



The navigation system simulator is the first ma jor'train- 
inq device USAF has procured for this system. Prior to its 
delivery, trainmcj was limited to providing introductory .train- 
ing, without the benefit of training equipment, at Keesler AFB 
and transferring graduates to^ an AWACS operational base for on- 
the-job training. ""The operational base represented the first 
hands-on experience 'received by the students. 

* Considering the continuity (and overlap) in the AWACS and 
P-J.6 programs, all three training systems should have extensive 
similarities in important features. Both the AWACS navigation 
trained and the F-16 systems employ Honeywell's Data Base 
IjGenerator, implying similarities in the software systems *(witb^ 
"which the generator^must be compatable). It is hard to escape 

sc. 

/ 
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a conclusion that 'these thref? programs have" provided Honeywell 
with the essentials of an advanced standardized simulation sys- 
tem that will f ind appficat ian in other training areas. 

Cost information on this program was provided by the AWACS 
Project office at the Electronics System Division (Table B-14K 
The original, FPIF contract with Honeywell provided for a target 
cost of $1274 thousand and a ce i 1 ing pr ice of $ 1528. One-en-- 
cjineerinq -change was negotiated for $60. thousand, bringing the 
total cost to the Government to $l r >88 thousand. 

y Thve contract has incurred a significant cost ovonun, 
Honeywell has reported costs „of roughly $200 thousand over the 
ceilihg price, but the Project Office believes the total over- 
run is\ ^proximately $600 thousand ,(or 40 percent of the ini- 
tial ceiling price). The Project Officer attributes the un- 
reported $400 thousand overrun to Honeywell's cost in develop- 
ing its I^ata ^ase Generator ( i\e, software) . (Seethe discus- 
sion of the F-16 program, above,) Development of the Data Base 
Generator v<as an independent HoneyweH decision, and its cost 
appears to hxive been charged to bqth the F-16 and* AWACS pro- 
gram^.) The\>r-dgram costs- shown in Table B-14 encompass the 
Project Office estimates of total costs (both reported and- un- 
reported). This is the only simulator proaram treated in this, 
fashion. Maintenance of the simulator is provided througn n 
separate FFP contract between Hoifeywell and the Air Force - 
Logistics Command (AFLC) . An estimate of Proiect Office in- 
house costs has not been obtained. 



TABLE B-14. AWA'CS NAV I GAT lON/Wl DANCE. >Ool 1 ars , Thousands) 



Contractor ^ * * 


American f 
Institute 

for UtuPfrrh 


Honeywell 

i 




• 

Task/Function ' 


Analysis 

and 
Program 
* Support 


Contract ,. 
with* ' 
System^ 
Program^ 
Office (ESD 


Maintenance 
Contract 
wUh 
) AFIC 


Total h 
Program" 

t 


"flon-Recurrinq Cost 

Front End Analysis 
Task Analysis 
Other * 


100 > 

i inn * 

< 100 \ 




f 


100/ * t 

1 100 


<0ts1gn and Development 
/ % Hardware". 

. Software/ Courseware 
Technical Data 
Other 




'1,998 
506 
943 

549 } 


< 


1,998 
506 
943 

549 W 


Hardware Fabrication * 

Test and Evaluation "» 
Program Management 
Total 'Non-Recurrlng 


* 

100 

200 • 


r 40 * 

23 
2,061 




r 

40 

* 1*3 
% 2,261 


Recurrinq Cost 

Production 
Hardware Fabrication ■ 

(recurring) 1 
Special Tools/Test 

tquip«<<ent 

Initial Spares 
Other 




127 
127 




127 
• 127 

S 


Logistic Support 
Interim Maintenance 

Support 
<$her 




70 . . i 


70 


• Initial Training 

Total Recurring 




i 


- _ _ — ^ 

197 ' 


Program Total Cost 

v. <. 

\ 


200 


2,188 


f i 

70 j 2,458 i 



•Including Installation and checkout 
Except for i^use program management for which estimate is not available. 
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APPENDJX C • 



■V AB^EVIATrdNS 

; A AACS, - Army Arfc 
K^-VS- l ABC \ m 4 Autpmatic 'Boiler eorttrpl 



W^'i}'^ ^ 4 A* s ' r* X ' " ^1 

* AACS/; # Army Arfca Communications Systlms 

Autpmatic 'Boiler Corttrpr ^ , . , . , 

Adaptive Computerized training ^ystem, (Peroeptron'ics) 
- * Autorriated- Data.* Processing- 
I Actual Equipment Trainer / !' ° **. ■ f'i 

Automated Electronics Maintenance .Trainer (Honeywell) 
^ Air Fqrce Human Resources Laboratory ^Brooks AFB)J 
Air Force Persohnei>and Training Research Gen tec i 
' %nbtt AFHRL) / V : '* 

fAir Fdjr^o Specif by Code / . ^ „ • 
Automated Instruction^ direction and' Exercise V 
. Avionics Intermediate Shop ^ , 
A : iT^ra;t/^aintenaftce effectiveness Simulation 




wy 

11 ; ' - 



A D P, 
ABT' 



'r~!$£yy a^ ; 

? " AFHRL 



If/ 



AFSC J 

Aroe^ 

ATS 
; AM'BS 



(a rnopSl, developed by XYZYX Information 
. 7 . • , f. : . Corporation fpr' NTBC) , ' 

•• >, ^' tf v'' ^'^vanced Manpower Cdncepts for Sea-^Based Aviation] 

/ ' Z. Automated Hai'nten|>nce Test Equipment 

xf v AHT^S^/, Army Maintenance Training and Evaiua 



}#.r. / . Af- 

ATACS^ 



4?" 



Syste|n,{ARI/PMTRADE) 
Ari§y-Navy 

Authorisation Parts List' 

Amy Research "institute : , 

Action Taken ' 

>Army Tactjcal Communications System 
Automatic Test Equipment 
Airbbrne Warning and Control System. 



Evaluation Simulation 



BIT, 
BITE 

CAI * 
CAM-T 

CASEE 

CB 

CIWS 

CMI 

CNR 

CNTT 

CPFP— 

CPIF 

CRS 

CRT 

CSD 

CUT 

DCS 
DRS 

DS ^ ' 
DSB 



ion 



tenance Training (Air Force 



Support Effectiveness 



Built-in Tfest „ 
> 

Built-in Test Equipment 

Computer-assisted Instruct 
Consolidated Aircraft Main 
hands-on. training) 
Comprehensiveness Aircraft 
Evaluation 
Component Breakdown 
Close-In Weapon System (Phialanx Gun\ System) 
Computer-Managed • Instruction 

Communication Navigation pladar 

I 

Chief of Naval Technical draining 
-Cost-plus— Fixed-Fee 1 — « 



i 



Cost Plus Incentive Fee 
Component Repair Squadron 

Cathode Ray Tube ■ . 

-I 

Constant Speed Drive ' / » 

/ 

Cross Utilization Trained (AF) 

/ / - 

Deputy Chief of Staff 
Detection and Ranging System , 
Direct Support 
Defense Science Board , 



ECC Educational Computer Corporation 

ECM Electronic Countermeasures 

EEMT Electronic Equipment Maintenance Training 

(Navy Class A Training! School , Honeywell) ' 

EPICS Enlisted Personnel Individualized Career Sysitem 

EIC Equipment Identification Code * 

ETM Extension Training Materials 



FCS 



•Fire Control System- 





^- ft-- 


- - 




- 1 




FFP 


_i 

Firm Fixed Price 






FIS 


Fault Identification Simulator (Navy) > ' 






FLIR 


Forward- Looking .Infrared 




| FOMM 


Functionally Oriented Maintenance Manual 






FPIF 

* 


Fixed Price Incentive Fee 




i ; 


FPJPA 


Fully Proceduralized Job Performance Aid 






FRAMP 


Fleet Readiness Aviation Maintenance Personnel 






FTD 


Field Trainipg Detachment (Air Force) 






FVS 


vFighting y^hicle System 






GMTS 


Generalized Maintenance Training System (Navy) 






GNS 


Guidance and Navigation System 






GS 


vj\» iic l q i oupuuL u 




> 


GSE 


Ground Support Equipment 






MIC 
HUD 

ICAP 
I HOMS 


Headquarters/Headquarters company 
He ads- up Display 

Improved Capability 

Intermediate Hands-on Maintenance Simulators ( NTEC) 




- 

-2 

> 


I level 

IMA 

IMTS 

INS 

IPSA 

TRR 

ISO 

I TOT 


Intermediate f jevel Maintenance 

Intermediate Maintenance Activity 

Integrated Maintenance- Training System (NTEC) 

Inertial Naviqation System 

Integrated Personnel Systems Approach 

Integrated Radio Room 

Instructional System Development 

integrated Technical Documentation and Training 

(Army? now callod SPAS) 


• 

- 

* 




JCN 


Job Control Number 






JPA ' 


Job Performance Aid 




* 


JTPT 


Job Task Performance Test 
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LAS Lockheed Aircraft Services 

LRU Line-Replaceabl e Un i t 

LTTA Logic Tree Troubleshooting Aid 

MACT Malfunction and Circuitry Trainer 

MAINTIP Maintenance Training Improvement Program (NTEC) 

MCAS Marine Corps Air Station 

MDC j Maintenance Dependency Chart 

MDS - Mode 1 /Des ignat ion/Ser ies (Air Force ) 

MFHBF Mean Flight Hours Between Failures 

MIL STD ' Military Standard 

MIMS Maintenance Instruction Manual System 

MITIPAC Modular Integration of Training Information by a 

Performance Aiding Computer (Navy) 

Mk/Mod Mark/Model 

MMTR Military Manpower Training Report 

MTTR Mean Time to Repair 

MOS Military Occupational Specialty (Army) 

MTBF Mean Time Between Failures 

MTBR Mean Time Between Repairs 

MTM Maintenance Training Management 

MTS Mobile Training Set (for Field Training Detachment) 

MTU Maintenance Training Unit 

NALCOMIS Navy Air Logistics Command Management Information 



System 

Naval Aviation Maintenance Program 

Naval Air Maintenance Training Detachment 

Naval Air Rework Facility 

Naval Air Station 

Naval Air Systems Command 

Navy Enlisted Classification * 

Not Fully Equipped 

NAMP Improvement Program 



NAMP" 
NAMTD 
NARF 
NAS 

NAVAIR 
NEC 
NFE 
NIP 
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NOR - Not Operationally Ready 

NORM -Not Operationally Ready - Maintenance *'" 
NORS Not Operationally Ready - Supply*. 

NPRDC Navy Personnel Research and Development Center 

NSN * -National Stock Number 

NSRDC David W. Tciylor Naval Ship Research and Development 

r- Center 
NTEC - Naval Training Equipment Center 

NTIPP. Navy Technical Information Presentation Program 

NTIPS ' Navy Technical Information Presentation System 

OJT On-the-job Training 

O level Organizational* Level Maintenance 



Pacific Aip Force 
Program Element 

Presentation of Information for Maintenance and 
Operation 

Performance Improvement for Navy Training Organizations 
Program Manager for Training Devices (Army) 
Production-Oriented Maintenance Organization ( AF) 
Programmable-Read-Only-Memory 

Reactive Electronic Equipment Simulator 
Request for Proposal 
Read-only Memory 

Resident Training Equipment (for technical training 
center) 

Support Action Code 

Simulated Avionicr Maintenance Trainer 
Sample Data Collect ; on 
Selected Equipment List 

Symbolic Integrated Maintenance Manual System 
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PACAF 
^PE 
PIMO 

PINTO 
, PMTRADE 
POMO 
PROM 

REES 
RFP 
ROM 
RTE 



SAC 

SAMT 

SDC 

SEL 

SIMMS 



SMART System Malfunction Analysis Reinf orcenent Trainer 

SMTE Simulated Maintenance Task Environment 

SOJT Supervised On-the-Job Training 

SPAS Skill Performance Aids System (Army; previously ITDT) 

SQT Skill Qualification Test (Army) 

SRA Shop-Replaceable Assembly 

SRU Shop-Replaceable Unit 

STRES Simulator Training Pequirements and Effectiveness 
Study (AFHRL) 

TAC Tactical Air' Command 

TACAN Tactical 'Air Navigation ^ 

TAMMS m The Army Maintenance Management System 

TICCIT Time-shared, Interactive Compute Controlled Informa- 
tion Television 

TJS Tactical Jamming System 

TM Type Maintenance 

TMS Type/Model/Series (Navy) 

TOT Task-Oriented Training 

TRAM Target Recognition Attack Multi-sensor * 

TRU Tester Replaceable Unit 

USAFE United States Air Force - Europe 

VAST Versatile Avionics Shop Test (Navy) 

VTAS Visual Target Acquisition System 

WBS Work Breakdown Structure 

WC Work Center 

WUC Work Unit Code 
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Maintenance and Material Management System (Navy) 



Air Force Maintenance Management System (name 
derived from th£ Air Force manual that sets forth 
maintenance policy.) \ 



APPENDIX D 
GLOSSARY 

Courseware: Student handbooks and manuals and that portion of 
the set of computer programs resident in a simulator that 
implements the simulation model and otherwise addresses the 
operation/functioning of the equipment being simulated* 

Cross-Skill Maintenance: Maintenance associated with one skill 
area that is performed by personnel trained in a different 
skill area* ^ 

Depots-Level Maintenance: Rear area maintenance, major repair 
or equipment modifications performed largely by civilians 
in military organizations* 

Direct-Support Maintenance: IntermejJiate-level maintenance per- 
formed in units attached to or organic with large combat 
units, e*g* , divisions (Army)* 

Fidelity: A normative term that describes the extent to which 

a sinulator duplicates its operational counterpart. Physi- 
cal fidelity refers to physical appearance, since a simu- 
lator may be two-dimensional or thre^dTmehsronn~l-r or- some 
combination, in its construction* Functional fidelity 
refers to the extent to which the performance characteris- 
tics of operational equipment have been duplicated in the 
simulator* 

General Support Maintenance: Intermediate-level maintenance 
performed in units attached to higher commands, e.g*, 
Corps, Theatre forces* 

Intermediate-Level Maintenance: Maintenance performed in a 
shop by a maintenance or repair unit* 
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Maintenance Action: All effort associated with the completion 
. of a maintenance requirement (e.g., the correction of a 

malfunction) that permits the return of equipment to an 

operational status. 
Maintenance Task: A single procedure that is performed as part 

of a maintenance action. For example, remove, troubleshoot 

repair, and install are cliscrete tasks m a maintenance 

action that corrects a malfunction |and restores equipment 

to operational status. 
Off-Equipment, Maintenance: Maintenance performed on equipment 

systems and assemblies that have been removed from weapon 

end- i terns . 

On-Pqui'pment Maintenance: Maintenance performed on equipment 
systems and assemblies while they are installed on weapon 
end- items. 

Organizational Level Maintenance: Maintenance performed directly 
oh operational equipment (e.q., fault detection, component 
replacement) by personnel assigned to units that operate 
the equipment. 

Simulation: The imitative representation of the operation/ 

functioninq of one system by 'another system. It consists 
of the simulation model, display and control panels, avrfTN. 
other input/output facilities peculiar to the system bkinq) 
simulated. r~ 

Simulation Model: A mathemat ical model that describes the \ 
operation/functioning of a particular system or equipment. / 
— S4mulato,rj_ The device (i.e., hardware and software) on which a/ 
simulation is imp 1 omen tec — - — ~~ — \ 

Simulator Model:- Simulators of a qiven (complete- or pdruaiT" ~~ 
conf iquration. 

Software: That portion of the set of computer programs resi- 
dent in a simulator that is not unique or peculiar to the 
system being simulated (i.e., the routines concerned with 
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utilities, input/output, translation, etc. that are em- 
ployed for general control of the computer) . 
Team Maintenance: Maintenance actions or tasks that are per- 
formed by more thati one person. 
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